Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
AI for Information Architects: Are the robots coming for our jobs?
Summary
Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.
Key Insights
-
•
AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.
-
•
Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.
-
•
Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.
-
•
Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.
-
•
Different AI models interpret the same content differently; model choice significantly affects results.
-
•
Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.
-
•
Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.
-
•
Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.
-
•
Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.
-
•
Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.
Notable Quotes
"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."
"You have to ask specific, very precise questions, not big general ones, to get useful AI results."
"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."
"Natural language processing is basically about figuring out how to math words."
"Different AI models have wildly different views of the same content; the model choice makes a big difference."
"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."
"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."
"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."
"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."
"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."
Or choose a question:
More Videos
"Community-rooted wisdom like the Mexican 'guana' practice reduces postpartum depression without traditional research."
Victor UdoewaBeyond Methods and Diversity: The Roots of Inclusion
March 26, 2024
"We often use radio buttons when check boxes or sliders would reveal the truth."
Peter MorvilleThe Architecture of Understanding
May 13, 2015
"When you mix trends together, you will see where change is likely to occur."
Sam LadnerHow Research Can Drive Strategic Foresight
March 9, 2022
"Having one centralized repository aligned with our culture and values made the most sense for us."
Michelle Bejian Lotia Anne-Marie MorellRolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
March 28, 2023
"Context is really super important not only for design and research but for how you design organizations and ways of working."
Leisa ReicheltOpening Keynote: Operating in Context
November 7, 2018
"The mayor wanted to shrink the neighborly bonds that the city was known for by investing in a 2030 living Monument of collaboration and community."
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"Our product simply sucks everything out of the open AI system and processes it with a knowledge graph based on ISO standards."
Joerg Beringer Thomas GeisScaling User Research with AI: Continuous Discovery of User Needs in Minutes
September 10, 2025
"It’s not enough to have great people; you need to create the space that brings out the best in them."
Peter MerholzCustomer-Centered Design Organizations
June 8, 2017
"Different people communicate in different ways, and understanding those ways gives you the key to convincing them."
Dalia El-ShimySo You've Got a Seat at the Table. Now What?
March 31, 2020