Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] How to re-categorize content at scale using LLMs
Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] How to re-categorize content at scale using LLMs
Speakers: Jorge Arango
Link:

Summary

Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.

Key Insights

  • Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.

  • Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.

  • An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.

  • Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.

  • A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.

  • GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.

  • The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.

  • Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.

  • The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.

  • Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.

Notable Quotes

"Some of the older content has discoverability problems, which is typical with blogs."

"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."

"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."

"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."

"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."

"A middle review step prevents hallucinations from making it into the production site."

"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."

"The entire process took about three hours from start to finish, about a fifth of the manual time."

"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."

"You need to review proposed changes before committing them to production, otherwise errors sneak in."

Ask the Rosenbot
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Justin Entzminger
Risk and Reward: How to Diversify the Field of Civic Innovators and Designers
2022 • Civic Design 2022
Gold
Harry Max
Failure Friday #5: Lessons from a SaaS Design Failure
2025 • Rosenfeld Community
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
Sheri Byrne-Haber
The Importance of Accessible Design Systems
2024 • DesignOps Summit 2020
Gold
Cassini Nazir
The Dangers of Empathy: Toward More Responsible Design Research
2023 • Advancing Research 2023
Gold
Aaron Stienstra
Leveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
2021 • Civic Design 2021
Gold
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Rich Mironov
How Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
2022 • Design in Product 2022
Gold
Darian Davis
Lessons from a Toxic Work Relationship
2024 • Enterprise Experience 2020
Gold
Séamus Byrne
Aligning Teams with Choreography
2024 • Enterprise Experience 2020
Gold
Louis Rosenfeld
Coffee with Lou
2024 • Rosenfeld Community
Sam Proulx
SUS: A System Unusable for Twenty Percent of the Population
2021 • Design at Scale 2021
Gold
Aurobinda Pradhan
Introduction to Collaborative DesignOps using Cubyts
2022 • DesignOps Summit 2022
Gold
Dave Gray
Group Activity: Making Sense of DesignOps
2017 • DesignOps Summit 2017
Gold
Michaela Mora
Advanced Concept Testing Approaches To Guide Product Development and Business Decisions
2022 • Advancing Research 2022
Gold

More Videos

Brian T. O’Neill

"If nobody uses this because they don’t trust it, it doesn’t matter how accurate the model is."

Brian T. O’Neill Maria Cipollone Luis Colin Manuel Dahm Mike Oren

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

February 18, 2022

Jorge Arango

"Large language models can help organize content faster and at much larger scale than people can."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024

Bria Alexander

"Sponsor sessions are not sales pitches; they are super similar to the quality of the main conference session."

Bria Alexander

Opening Remarks

October 1, 2021

Louis Rosenfeld

"Our curators have been working for months taking presentations from little ideas to something big that advances design ops as practice."

Louis Rosenfeld Bria Alexander

Day 1 Welcome

September 23, 2024

Mansi Gupta

"How can we not forget about women?"

Mansi Gupta

Women-Centric Research: What, Why, How

March 29, 2023

Bria Alexander

"Sponsor sessions do not overlap at all with the main programming. It is 100% fully optional."

Bria Alexander

Opening Remarks

October 3, 2023

Andy Polaine

"If you are comfortable with it, put your cameras on. It makes me not feel like we’re talking to an answering machine."

Andy Polaine Lavrans Løvlie

What is the role of service design in product-led organizations?

December 3, 2024

George Abraham

"Publishing again creates a new branch with changes and a pull request, making collaboration and version control seamless."

George Abraham Stefan Ivanov

Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)

October 1, 2021

Robin Beers

"Building coalitions and embedding research insights within the culture is essential for long-term impact."

Robin Beers

Beyond Insights: Researchers as Organizational Change Catalysts

March 25, 2024