Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI for Information Architects: Are the robots coming for our jobs?

Thursday, November 21, 2024 • Rosenfeld Community
Share the love for this talk
AI for Information Architects: Are the robots coming for our jobs?
Speakers: Karen McGrane and Jeff Eaton
Link:

Summary

Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.

Key Insights

  • AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.

  • Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.

  • Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.

  • Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.

  • Different AI models interpret the same content differently; model choice significantly affects results.

  • Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.

  • Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.

  • Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.

  • Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.

  • Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.

Notable Quotes

"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."

"You have to ask specific, very precise questions, not big general ones, to get useful AI results."

"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."

"Natural language processing is basically about figuring out how to math words."

"Different AI models have wildly different views of the same content; the model choice makes a big difference."

"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."

"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."

"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."

"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."

"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."

Ask the Rosenbot
Louis Rosenfeld
The Rosenbot and the Rosenverse: An AMA with Lou Rosenfeld
2024 • Designing with AI 2024
Gold
Uday Gajendar
From AI to Zeitgeist: Theory as the design antidote to AI hype
2025 • Rosenfeld Community
Theresa Neil
Designing for Wellness: Specializing in Healthcare
2024 • Rosenfeld Community
Sol Mesz
Hands or Brains? How to Hire for Strategy, Strategically
2024 • Enterprise Experience 2020
Gold
Sheryl Cababa
Expanding your Design Lens with Systems Thinking
2023 • Advancing Research 2023
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • Civic Design 2021
Gold
Samuel Proulx
Inclusive Research: Debunking Myths and Getting Started
2025 • Advancing Research 2025
Gold
Russell Blair
Killing the blank page
2024 • Designing with AI 2024
Gold
Tricia Wang
Spatial Collapse: Designing for Emergent Culture
2024 • Enterprise Experience 2020
Gold
John Donmoyer
Shipping your code generation experiments to production
2025 • Designing with AI 2025
Gold
Louis Rosenfeld
How AI will Change DesignOps Tooling
2023 • DesignOps Summit 2023
Gold
Roberta Dombrowski
Making Research a Team Sport
2022 • Advancing Research 2022
Gold
Chris Moses
Stretching the Definition of DesignOps with Product Development
2018 • DesignOps Summit 2018
Gold
Jon Temple
Panel: Stacks, Security, and Stakeholders: The Hidden Work of UXR Tool Procurement
2026 • Advancing Research 2026
Gold
Connect with the Advancing Research Community
2026 • Advancing Research 2026
Gold
Sam Proulx
Everything You Ever Wanted to Know About Screen Readers
2021 • Design at Scale 2021
Gold

More Videos

Caroline Vize

"User experience metrics can give you a well-rounded perspective beyond mere site analytics."

Caroline Vize

The State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022

March 9, 2022

Kara Kane

"Having a monthly newsletter gave people one place to go instead of having to check Slack or emails all the time."

Kara Kane

Communities of Practice for Civic Design

April 7, 2022

Jon Fukuda

"We don’t need a north star, we need a constellation that allows us to see the full picture."

Jon Fukuda Ellie Krysl

Design Planning and Management Support

October 3, 2023

Mark Templeton

"You can leave a legacy in three minutes by giving a connection or sharing an insight—it doesn’t have to be grand."

Mark Templeton

Creating a Legacy: the ultimate experience

June 9, 2017

Jim Kalbach

"Experience mapping brought a new understanding that helped us draw former extremists into peace efforts."

Jim Kalbach

Peace is waged with sticky notes: Mapping Real-World Experiences

June 14, 2018

Scott Stephens

"You can connect tools like Zapier or Integromat to Coda packs to automate processes and integrate with external systems."

Scott Stephens

The Next Generation in DesignOps Toolsets

July 28, 2022

Cheryl Platz

"Low status looks like slumping, shuffling feet, and avoiding eye contact but seeking approval."

Cheryl Platz

Collaborative Creativity through Improv

November 7, 2018

Jen Briselli

"We don’t stand back to reflect, but we can do that reflecting in the moment."

Jen Briselli

Learning Is The Engine: Designing & Adapting in a World We Can’t Predict

April 16, 2025

Mac Smith

"We set a two-year goal to advise leadership on top issues for annual planning, even though we hadn’t figured out all the steps."

Mac Smith

Measuring Up: Using Product Research for Organizational Impact

March 12, 2021