Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Megan Blocker
Getting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
2024 • Advancing Research 2024
Gold
Gabriela Barneva
Operationalizing Inclusive Design in Service Design
2025 • Advancing Service Design 2025
Gold
Doug Powell
DesignOps and the Next Frontier: Leading Through Unpredictable Change
2025 • DesignOps Summit 2025
Gold
Andy Warr
Under My (Research) Umbrella: The Benefits and Challenges of Building a Unified Insights Function
2024 • Advancing Research 2024
Gold
Francesca Barrientos, PhD
You Need Your Own Definition of Design Maturity
2022 • Design at Scale 2022
Gold
Nicole Aleong
Future Orientations to Everyday Life: Futures Anthropology as a Methodology
2024 • Advancing Research 2024
Gold
Ned Dwyer
Right horses for the right courses – how and when to democratize research
2025 • Advancing Service Design 2025
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Stefanie Owens
Optimizing for Outcomes: Transformation Design in Systems at Scale
2024 • Advancing Service Design 2024
Gold
Tim Frick
The journey of building a sustainable design practice
2025 • Climate UX Interest Group
Cara Maritz
The Art of Extrapolation
2023 • Advancing Research 2023
Gold
Marc Fonteijn
Increase your confidence, influence, and impact (through a Professional Community)
2024 • Advancing Service Design 2024
Gold
Uday Gajendar
From AI to Zeitgeist: Theory as the design antidote to AI hype
2025 • Rosenfeld Community
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Dave Malouf
Panel: Design Systems and Documentation
2017 • DesignOps Summit 2017
Gold
John Maeda
Making Sense of Enterprise UX
2016 • Enterprise UX 2016
Gold

More Videos

Helen Armstrong

"The relationship between designers and data scientists can actually be pretty magical."

Helen Armstrong

Augment the Human. Interrogate the System.

June 7, 2023

Christian Bason

"The New European Bauhaus combines aesthetics, inclusivity, and climate-neutral ambitions to transform built environments by 2050."

Christian Bason

Expand—Rethinking Design for Public Challenges

September 14, 2022

Onur Kocan

"Prost's master plan imposed a heavy interventionist burden on the historical structure of the city."

Onur Kocan Ayhan Ensici

Understanding the Strategy for Civic Design in a Complex City: Istanbul

November 16, 2022

Peter Van Dijck

"It’s much easier to classify an answer than to generate an answer, and that’s why LLM as a judge works."

Peter Van Dijck

Building impactful AI products for design and product leaders, Part 2: Evals are your moat

July 23, 2025

Nalini Kotamraju

"What happens if research does not report into UX? What might happen if we uncouple research from UX?"

Nalini Kotamraju

Research After UX

March 25, 2024

Brendan Jarvis

"Are the questions that you are asking and the stories that you are telling yourself serving you?"

Brendan Jarvis

Framing Tomorrow by Questioning Today

June 8, 2022

Bria Alexander

"Our job as civic designers is to bring government needs and community needs into alignment."

Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne Dong

Reflect and Chart Forward

December 10, 2021

Frances Yllana

"Leaders should look for ways to get firsthand experience with these tools, not just rely on media."

Frances Yllana Jorge Arango Maria Taylor Briana Thomas

The Big Question about Impact: A Panel Discussion

September 24, 2024

Liz Ebengo

"Hope is not passive. Hope is very proactive and requires facilitation that helps people dream despite their harsh realities."

Liz Ebengo

The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward

December 4, 2024