Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI for Information Architects: Are the robots coming for our jobs?
Thursday, November 21, 2024 • Rosenfeld Community
Share the love for this talk
AI for Information Architects: Are the robots coming for our jobs?
Speakers: Karen McGrane and Jeff Eaton
Link:

Summary

Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.

Key Insights

  • AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.

  • Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.

  • Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.

  • Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.

  • Different AI models interpret the same content differently; model choice significantly affects results.

  • Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.

  • Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.

  • Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.

  • Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.

  • Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.

Notable Quotes

"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."

"You have to ask specific, very precise questions, not big general ones, to get useful AI results."

"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."

"Natural language processing is basically about figuring out how to math words."

"Different AI models have wildly different views of the same content; the model choice makes a big difference."

"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."

"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."

"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."

"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."

"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."

Ask the Rosenbot
Dan Saffer
Why AI projects fail (and what we can do about it)
2025 • Rosenfeld Community
Trisha Causley
[Demo] Complexity in disguise: Crafting experiences for generative AI features
2024 • Designing with AI 2024
Gold
Teresa Swingler
Look, Up in the Sky! UX/UI for Aerospace
2022 • Enterprise Community
Louis Rosenfeld
Do you want to work on climate? (Climate UX Discussion Series)
2023 • Climate UX Interest Group
Uday Gajendar
10 Years of Enterprise UX: Reflecting on the community and the practice
2025 • Enterprise Community
Victor Udoewa
Theme One Intro
2023 • Advancing Research 2023
Gold
Terry Buckman
Wargaming (An Introduction)
2023 • Enterprise Community
Jeff Gothelf
Innovation Studios: the Engines of Enterprise Experimentation
2015 • Enterprise UX 2015
Gold
Candace Myers
Standardizing Design at Scale
2022 • DesignOps Summit 2022
Gold
Molly Fargotstein
Multipurpose Communication & UX Research Marketing
2019 • DesignOps Community
Ovetta Sampson
Turning UX Passion into Real Product Influence
2023 • Enterprise UX 2023
Gold
George Hinchliffe
Delivering Amazing Experiences
2021 • Design at Scale 2021
Gold
Meredith Black
Scaling Design Culture
2017 • DesignOps Summit 2017
Gold
Sarah Rink
Remote User Research: Dos and Don'ts from the Virtual Field
2020 • Advancing Research Community
Laura Gatewood
Beyond Buzzwords: Adding Heart to Effective Slack Communication
2024 • DesignOps Summit 2024
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold

More Videos

Bria Alexander

"This day is focused on me, my job and my specific day-to-day challenges, not on some invisible finish line or senior title."

Bria Alexander

Theme Two Intro

October 3, 2023

Brad Peters

"I try as a product manager to involve them in those UX practices as often as possible."

Brad Peters Anne Mamaghani

Short Take #1: UX/Product Lessons from Your Industry Peers

December 6, 2022

Bria Alexander

"Slack is the heart and soul of this conference — that’s where you ask questions, get help, and connect with others."

Bria Alexander

Opening Remarks

September 9, 2022

Nancy Douyon

"People in some countries are three times more likely to buy a product if it’s localized in their language."

Nancy Douyon

We'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex

June 15, 2018

Roberta Dombrowski

"Storytelling isn’t just for qualitative research; it’s also powerful for framing quantitative data to drive decisions."

Roberta Dombrowski Sam Duong Woloszynski

Making Research a Team Sport

March 11, 2022

Prayag Narula

"The focus is on helping teams take better decisions, not handholding them through every step."

Prayag Narula Abhinav Krishna

Dialing for Research: How to Reach the Unreachable

March 10, 2022

Tom Armitage

"There’s a reason we’re always taught to write notes or when we revise, we write notes out by hand, the hand remembers."

Tom Armitage Carla Diana Kanene Ayo Holder

Day 2 Panel: Looking ahead: Designing with AI in 2026

June 11, 2025

Jeff Ephraim Bander

"Subjective user ratings said a task was fine, but eye tracking showed they didn’t see important information."

Jeff Ephraim Bander Ariane Rahn Philipp Reiter

Eye Tracking Gamechanger: Why Smartphone Eye Tracking will Revolutionize Your UX Research

March 11, 2022

Amy Paris

"Government forms need to represent identity and respect privacy so we can better serve all people."

Amy Paris Danielle Thierry

Delivering Equity: Government Services for All Ages, Languages, Sexual Orientations, and Gender Identities

December 9, 2021