Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] How to re-categorize content at scale using LLMs

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] How to re-categorize content at scale using LLMs
Speakers: Jorge Arango
Link:

Summary

Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.

Key Insights

  • Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.

  • Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.

  • An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.

  • Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.

  • A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.

  • GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.

  • The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.

  • Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.

  • The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.

  • Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.

Notable Quotes

"Some of the older content has discoverability problems, which is typical with blogs."

"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."

"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."

"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."

"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."

"A middle review step prevents hallucinations from making it into the production site."

"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."

"The entire process took about three hours from start to finish, about a fifth of the manual time."

"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."

"You need to review proposed changes before committing them to production, otherwise errors sneak in."

Ask the Rosenbot
Edgar Anzaldua Moreno
Using Research to Determine Unique Value Proposition
2021 • Advancing Research 2021
Gold
Kyria Stephens
Power to Heal: Civic Design in the Aftermath of Tragedy
2022 • Civic Design 2022
Gold
Jorge Arango
Exploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)
2024 • Rosenfeld Community
Kristin Taylor
Building Bridges Across Organizational Silos
2022 • Civic Design 2022
Gold
Sam Ladner
Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX
2016 • Enterprise UX 2016
Gold
Jennifer Kong
Journeying toward AI-assisted documentation in healthcare
2024 • Designing with AI 2024
Gold
Jen Briselli
Learning Is The Engine: Designing & Adapting in a World We Can’t Predict
2025 • Rosenfeld Community
Jamika Burge
Embracing change: Navigating shifting landscapes with compassion and agency
2025 • Advancing Research 2025
Gold
Dr. Jamika D. Burge
Advancing the Inclusion of Womxn in Research Practices
2022 • Advancing Research Community
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Civic Design 2022
Gold
Cheryl Platz
Collaborative Creativity through Improv
2018 • DesignOps Summit 2018
Gold
Sabrina Mach
How to Design Your Design Operating Model
2021 • DesignOps Summit 2021
Gold
Jen Briselli
Learning is the north star: service design for adaptive capacity
2025 • Advancing Service Design 2025
Gold
Meghan Bausone
Systems Thinking and Design Innovation: Working with Leverage Points in Rural Maternal Health Systems
2026 • Rosenfeld Community
Dan Mall
“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales
2023 • DesignOps Summit 2023
Gold
Gabriela Barneva
Operationalizing Inclusive Design in Service Design
2025 • Advancing Service Design 2025
Gold

More Videos

Sarah Coyle

"We are just starting to do regular and frequent reporting across the entire design organization."

Sarah Coyle

Design and Analytics with Sarah Coyle

July 30, 2020

Kit Unger

"Teams only think about a specific customer obsession persona and lose sight of upstream or downstream effects and other personas."

Kit Unger Jackie Ho Veevi Rosenstein Vasileios Xanthopoulos

Theme 2: Discussion

January 8, 2024

Bria Alexander

"There is no overlap between the main program sessions and sponsor sessions."

Bria Alexander

Opening Remarks

January 8, 2024

Mike Oren

"Organizations are slow to adopt the practice of making every decision from the voice of the customer, despite having talented people and data."

Mike Oren Janice Wiitala

Design Research Strategy & Strategic Design Research

February 3, 2022

Mackenzie Guinon

"I call this the moment of I’ve been doing this all along—people realize they were practicing research before they knew the name."

Mackenzie Guinon

M.C. Escher’s UX Research Career Ladder

March 9, 2022

Lada Gorlenko

"Taking care of ourselves first is essential before we can effectively care for others."

Lada Gorlenko

Theme 3: Introduction

June 10, 2021

Matt Duignan

"Curation is super important, but also super hard."

Matt Duignan

Atomizing Research: Trend or Trap

March 30, 2020

Robert Fabricant

"We are people who are chosen — our craft is to interact with other people, and that’s not what we’re seeing in the world around us."

Robert Fabricant

Shifting dynamics: The evolving relationship between researchers, participants, and organizational systems

March 11, 2025

Alba Villamil

"Culture is the means by which people understand or make sense of the world around them."

Alba Villamil

Stereotyped by Design: Pitfalls in Cross-Cultural User Research

March 30, 2020