Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Molly Fargotstein
Multipurpose Communication & UX Research Marketing
2019 • DesignOps Community
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Kavana Ramesh
Meaningful inclusion: Practicing accessibility research with confidence
2024 • DesignOps Summit 2024
Gold
Jemma Ahmed
Research at an inflection point: Adapting to a new era of collaboration, equity, and innovation
2025 • Advancing Research 2025
Gold
Dan Willis
Enterprise Storytelling Sessions
2019 • Enterprise Experience 2019
Gold
Ian Johnson
Latent Scope: Finding structure in unstructured data
2025 • Designing with AI 2025
Gold
Fisayo Osilaja
[Demo] The AI edge: From researcher to strategist
2024 • Designing with AI 2024
Gold
Kristin Skinner
Theme 2: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Candace Myers
Standardizing Design at Scale
2022 • DesignOps Summit 2022
Gold
Ruzanna Rozman
Getting in Flow with Your Team
2024 • DesignOps Summit 2020
Gold
Jon Fukuda
The Big Question about Innovation: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Gina Mendolia
Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
2024 • Advancing Service Design 2024
Gold
Dan Mall
“Ask Me Anything” with Dan Mall, Author of Upcoming Rosenfeld Title, Design that Scales
2023 • DesignOps Summit 2023
Gold
Uday Gajendar
The Wicked Craft of Enterprise UX
2015 • Enterprise UX 2015
Gold
Jim Kalbach
Jazz Improvisation as a Model for Team Collaboration
2017 • DesignOps Summit 2017
Gold
George Aye
That Quiet Little Voice: When Design and Ethics Collide
2022 • Civic Design 2022
Gold

More Videos

Brian T. O’Neill

"Smoke comes out of my ears when I'm talking to some of the more advanced applications in machine learning."

Brian T. O’Neill Maria Cipollone Luis Colin Manuel Dahm Mike Oren

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

February 18, 2022

Jorge Arango

"I used GPT-4 to retag 1,200 blog posts, reducing an estimated 10–12 hours of tedious manual work to about a third of that time."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024

Bria Alexander

"Our community would not be what it is without you."

Bria Alexander

Opening Remarks

October 1, 2021

Louis Rosenfeld

"Please put your questions in the Slack thread associated with the talk, not just the general channel, or they’ll get lost."

Louis Rosenfeld Bria Alexander

Day 1 Welcome

September 23, 2024

Mansi Gupta

"Women are living nonlinear lives—career breaks, health cycles—that standard designs do not account for."

Mansi Gupta

Women-Centric Research: What, Why, How

March 29, 2023

Bria Alexander

"Sponsors bring so much color and so much light into the community."

Bria Alexander

Opening Remarks

October 3, 2023

Andy Polaine

"Lots of service design has been happening but in disguise and not called that."

Andy Polaine Lavrans Løvlie

What is the role of service design in product-led organizations?

December 3, 2024

George Abraham

"We want to leapfrog over individually inspecting things; developers won't have to copy and paste anything."

George Abraham Stefan Ivanov

Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)

October 1, 2021

Robin Beers

"We are the only thing we can control, how quickly we learn and how we adapt based on that learning."

Robin Beers

Beyond Insights: Researchers as Organizational Change Catalysts

March 25, 2024