Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.
Key Insights
-
•
AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.
-
•
Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.
-
•
Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.
-
•
Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.
-
•
Different AI models interpret the same content differently; model choice significantly affects results.
-
•
Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.
-
•
Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.
-
•
Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.
-
•
Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.
-
•
Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.
Notable Quotes
"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."
"You have to ask specific, very precise questions, not big general ones, to get useful AI results."
"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."
"Natural language processing is basically about figuring out how to math words."
"Different AI models have wildly different views of the same content; the model choice makes a big difference."
"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."
"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."
"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."
"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."
"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."
Or choose a question:
More Videos
"We made this move in January 2020 — our research and insights team moved out of UX but stayed within product organizations."
Nalini KotamrajuResearch After UX
March 25, 2024
"If you want to go fast, go alone; if you want to go far, go together."
Dean BroadleyNot Black Enough to be White
January 8, 2024
"Silence is complicity. Inaction is support."
Denise Jacobs Nancy Douyon Renee Reid Lisa WelchmanInteractive Keynote: Social Change by Design
January 8, 2024
"Design thinking helped us empathize, define problems, co-create ideas, and then test them like products."
Kim Fellman CohenMeasuring the Designer Experience
October 23, 2019
"Power is the ability to change another person’s reality."
George AyeThat Quiet Little Voice: When Design and Ethics Collide
November 16, 2022
"I don't want a component library for the account home. I want a system for all of that."
Nathan CurtisBeyond the Toolkit: Spreading a System Across People & Products
June 9, 2016
"Real estate transactions are among the top five life events in terms of emotional significance for people."
Greg PetroffThe Compass Mission
March 10, 2021
"Global and local cultures are closely entwined and always influencing one another."
Chloe Amos-EdkinsA Cultural Approach: Research in the Context of Glocalisation
March 27, 2023
"Having the qualitative and quantitative data together lets us tell a comprehensive story to stakeholders."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022