Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Denise Jacobs
Interactive Keynote: Social Change by Design
2024 • Enterprise Experience 2020
Gold
Margot Bloomstein
Fostering Trust in Your Brand and Beyond
2020 • Enterprise Community
Elizabeth Churchill
Exploring Cadence: You, Your Team, and Your Enterprise
2017 • Enterprise Experience 2017
Gold
Craig Brookes
"Just Make it Look Good" and Other Ways We're Misunderstood
2021 • Design at Scale 2021
Gold
Mandy Drew
What Role(s) Can Research Play in Responsible Design?
2021 • Advancing Research 2021
Gold
Surya Vanka
Unleashing Swarm Creativity to Solve Enterprise Challenges
2021 • Design at Scale 2021
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • DesignOps Summit 2025
Conference
Brendan Jarvis
Framing Tomorrow by Questioning Today
2022 • Design at Scale 2022
Gold
Luca Rager
Empowering Gaming at Scale: How Xbox Builds Powerful, Automated, and Distributed Design Systems with Sketch
2021 • DesignOps Summit 2021
Gold
Maria Skaaden
Continuous Design: One eye on the horizon and the other on the next wave
2018 • DesignOps Summit 2018
Gold
David Cronin
The GE Design System and Thoughts about Craft at Scale
2015 • Enterprise UX 2015
Gold
Katie Hansen
Finding the unknown in the known: Harnessing meta-analysis and literature review
2025 • Advancing Research 2025
Gold
Scher Foord
Turn the Ship Around: How to Apply Design Thinking Across Your Organization
2021 • Design at Scale 2021
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Advancing Research 2023
Gold

More Videos

Jon Fukuda

"User research is really only part of the information schema that we’re talking about."

Jon Fukuda Ellie Krysl

Design Planning and Management Support

October 3, 2023

Saara Kamppari-Miller

"Representing a person as a circle feels more right—like we’re all equidistant at the same table."

Saara Kamppari-Miller

Cartography for Design Communities

September 10, 2025

Kevin Bethune

"Everything is the way it is by design."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Daniel Orbach

"Process is the scaffolding that helps culture grow when done right."

Daniel Orbach

Zero to One: Co-Creating Operating Models with your Team

September 23, 2024

Marc Fonteijn

"Half of the service design professionals do not feel connected to their peers, which is significant."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Clara Kliman-Silver

"AI is valuable in design when it extrapolates away mundane work."

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Robin Beers

"The table’s not a one-time performance; it’s multimodal with before and after moments you can use."

Robin Beers Nalini Kotamraju Andy Warr

Panel: Excellence in Communicating Insights

March 26, 2024

Ana Maria Montero Barrantes

"Puppetry lets you communicate like, I don’t like what you said, how come you said that to me."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Kristen Honey

"Long COVID will teach us so much about invisible illnesses that are common but overlooked."

Kristen Honey

"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design

November 18, 2021