Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] How to re-categorize content at scale using LLMs
Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] How to re-categorize content at scale using LLMs
Speakers: Jorge Arango
Link:

Summary

Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.

Key Insights

  • Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.

  • Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.

  • An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.

  • Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.

  • A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.

  • GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.

  • The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.

  • Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.

  • The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.

  • Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.

Notable Quotes

"Some of the older content has discoverability problems, which is typical with blogs."

"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."

"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."

"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."

"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."

"A middle review step prevents hallucinations from making it into the production site."

"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."

"The entire process took about three hours from start to finish, about a fifth of the manual time."

"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."

"You need to review proposed changes before committing them to production, otherwise errors sneak in."

Cara Maritz
The Art of Extrapolation
2023 • Advancing Research 2023
Gold
Cheryl Platz
Collaborative Creativity through Improv
2018 • DesignOps Summit 2018
Gold
Joshua Graves
We Need To Talk: Navigating Conversations with Your Boss (Part 1 of 3)
2025 • Rosenfeld Community
Chris Geison
What is Research Strategy?
2021 • Advancing Research 2021
Gold
Angelos Arnis
State of DesignOps: Learnings from the 2021 Global Report
2021 • DesignOps Summit 2021
Gold
Patrizia Bertini
Pushing DesignOps’ Influence into New Global Markets
2022 • DesignOps Summit 2022
Gold
Ovetta Sampson
Turning UX Passion into Real Product Influence
2023 • Enterprise UX 2023
Gold
Corey Nelson
Layoffs
2022 • Advancing Research Community
Kit Unger
Theme 2: Discussion
2024 • Enterprise Experience 2020
Gold
Jamika Burge
Embracing change: Navigating shifting landscapes with compassion and agency
2025 • Advancing Research 2025
Gold
Ivana Ng
Level Up Your Program with ProductOps
2024 • Enterprise Experience 2020
Gold
Wendy Johansson
An Education on Design Education for Orgs
2021 • Design at Scale 2021
Gold
Dave Hoffer
UX Job Search AMA #3 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Saara Kamppari-Miller
DesignOps for Inclusive Design and Accessibility
2022 • DesignOps Community
Rachael Dietkus, LCSW
Everything You Need to Know about the Civic Design 2022 Call for Presentations
2022 • Civic Design Community
Sarit Geertjes
People, not Petri Dishes: Stories from a Research Recruiter
2019 • DesignOps Community

More Videos

Yoel Sumitro

"Students cannot be taught what they need to know; they can only be coached to absorb it through demonstration."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Zariah Cameron

"We need to move the speed of trust. If I don’t trust you, how can I feel safe with you?"

Zariah Cameron

ReDesigning Wellbeing for Equitable Care in the Workplace

September 23, 2024

Doug Powell

"We make the invisible visible and turn abstract strategy into something people can see, touch, and act on."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Darian Davis

"I felt on edge and physically depleted after our interactions."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Mike Oren

"To innovate smarter, you need to get access to the roadmap as early as possible and start research even when not asked for it."

Mike Oren

Why Pharmaceutical's Research Model Should Replace Design Thinking

March 28, 2023

Jacqui Frey

"Scale is social work because it’s about creating relationships and reducing friction."

Jacqui Frey

Scale is Social Work

March 19, 2020

Kim Holt

"If you find someone not following values, start with a one-on-one to understand their point of view and realign together."

Kim Holt Emma Wylds Pearl Koppenhaver Maisee Xiong

A Salesforce Panel Discussion on Values-Driven DesignOps

September 8, 2022

Samuel Proulx

"Accessibility is innovation and the kinds of features people with disabilities need are incredible conveniences for the rest of us."

Samuel Proulx

From Standards to Innovation: Why Inclusive Design Wins

September 10, 2025

Laine Riley Prokay

"Gender-coded words influence someone’s decision to apply and feel they belong even within internal career ladders."

Laine Riley Prokay

How DesignOps can Drive Inclusive Career Ladders for All

September 30, 2021