Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] How to re-categorize content at scale using LLMs
Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] How to re-categorize content at scale using LLMs
Speakers: Jorge Arango
Link:

Summary

Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.

Key Insights

  • Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.

  • Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.

  • An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.

  • Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.

  • A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.

  • GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.

  • The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.

  • Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.

  • The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.

  • Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.

Notable Quotes

"Some of the older content has discoverability problems, which is typical with blogs."

"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."

"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."

"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."

"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."

"A middle review step prevents hallucinations from making it into the production site."

"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."

"The entire process took about three hours from start to finish, about a fifth of the manual time."

"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."

"You need to review proposed changes before committing them to production, otherwise errors sneak in."

Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Ted Booth
Discussion
2016 • Enterprise UX 2016
Gold
Dan Hill
Designing for the infrastructures of everyday life
2024 • Designing with AI 2024
Gold
Sarah Rink
Remote User Research: Dos and Don'ts from the Virtual Field
2020 • Advancing Research Community
Theresa Marwah
How Atlassian is Operationalizing Respect in Research
2020 • Advancing Research Community
Bob Baxley
Theme 4: Intro
2024 • Enterprise Experience 2020
Gold
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Bria Alexander
Welcome
2022 • DesignOps Summit 2022
Gold
Rachael Dietkus, LCSW
Trauma-Responsive Design: Reimagining the Future of Design Now
2021 • Civic Design 2021
Gold
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Rachael Dietkus, LCSW
The power to heal and harm
2025 • Advancing Research 2025
Gold
Megan Blocker
A Selectively Scrappy Approach to ResearchOps
2018 • DesignOps Summit 2018
Gold
Tricia Wang
Spatial Collapse: Designing for Emergent Culture
2024 • Enterprise Experience 2020
Gold
Lisanne Norman
Why I Left Research
2023 • Advancing Research 2023
Gold
Joi Freeman
A New Vantage Point: Building a Pipeline for Multifaceted Research(ers)
2020 • Advancing Research 2020
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold

More Videos

Jon Fukuda

"We don’t need a north star, we need a constellation that allows us to see the full picture."

Jon Fukuda Ellie Krysl

Design Planning and Management Support

October 3, 2023

Saara Kamppari-Miller

"Representing a person as a circle feels more right—like we’re all equidistant at the same table."

Saara Kamppari-Miller

Cartography for Design Communities

September 10, 2025

Kevin Bethune

"Everything is the way it is by design."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Daniel Orbach

"Sometimes organic culture pieces, like playing Wordle at the end of standup, shorten meetings and keep them tight."

Daniel Orbach

Zero to One: Co-Creating Operating Models with your Team

September 23, 2024

Marc Fonteijn

"A regular heartbeat and familiar structure are really important principles to keep a community alive."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Clara Kliman-Silver

"Novice users can try things out thanks to automation, but experts can focus on the challenging and creative pieces."

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Robin Beers

"Make sure your leadership team knows your name."

Robin Beers Nalini Kotamraju Andy Warr

Panel: Excellence in Communicating Insights

March 26, 2024

Ana Maria Montero Barrantes

"I am not throwing away your UX."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Kristen Honey

"It’s easier to create a better system that makes your old system obsolete than to change what is baked in."

Kristen Honey

"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design

November 18, 2021