Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Jemma Ahmed
Democratization: Working with it, not against it [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Scott Stephens
The Next Generation in DesignOps Toolsets
2022 • DesignOps Community
Anna Nguyen
Why Our Voice of the Customer is Better Than Yours
2022 • Advancing Research 2022
Gold
Dave Malouf
The Future of DesignOps
2024 • DesignOps Summit 2020
Gold
Mike Brzozowski
UX in everyday products: Empowering climate conscious choices
2024 • Climate UX Interest Group
Crystal Philcox
The Many Faces of Operations
2017 • DesignOps Summit 2017
Gold
Cennydd Bowles
Exit Interview #2: Rediscovering the ethical heart of design
2025 • Rosenfeld Community
Megan Clegg
Space for Everyone: Reframing Accessibility Through a Wider Lens
2021 • Design at Scale 2021
Gold
Matt Webb
Context Window: Five Futures for AI
2025 • Designing with AI 2025
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Anne Mamaghani
How Your Organization's Generative Workshops Are Probably Going Wrong and How to Get Them Right
2023 • Advancing Research 2023
Gold
Connect with the Advancing Research Community
2026 • Advancing Research 2026
Conference
Laura Schaefer
DesignOps: A Conduit for Inclusion
2022 • DesignOps Summit 2022
Gold
Lija Hogan
Practical Principles of Inclusive Research
2023 • Advancing Research 2023
Gold
Trisha Terhar
Empathizing with the Empowered: Non-Researcher Responses to Democratization
2022 • Advancing Research 2022
Gold

More Videos

Sam Proulx

"My screener automatically reads out all incoming messages into my headphones, so I hear my voice, the Zoom messages, the speaker notes, and the slides all at once."

Sam Proulx

Online Shopping: Designing an Accessible Experience

November 29, 2023

Farid Sabitov

"Speed and quality are the new competitive advantages; doing the right thing is no longer enough."

Farid Sabitov

Automatization for Large Enterprise Teams

January 8, 2024

Chloe Amos-Edkins

"Global and local cultures are closely entwined and always influencing one another."

Chloe Amos-Edkins

A Cultural Approach: Research in the Context of Glocalisation

March 27, 2023

Sam Proulx

"People said graphical user interfaces are bad for accessibility and that blind people would have to stick to command lines forever."

Sam Proulx

To Boldly Go: The New Frontiers of Accessibility

September 9, 2022

Sarah Auslander

"Small, tactical projects helped us secure political will and organizational capacity needed for systemic policy change."

Sarah Auslander

Incremental Steps to Drive Radical Innovation in Policy Design

November 18, 2022

Peter Van Dijck

"Evals define a shared definition of good with tests to measure it, and that is the secret sauce for building great AI products."

Peter Van Dijck

Hands-on AI #2: Understanding evals: LLM as a Judge

October 15, 2025

Maish Nichani

"The service center redesign with IDEO turned a crowded, intimidating place into a five-star hotel experience with name calls and family cabanas."

Maish Nichani

Sparking a Service Excellence Mindset at a Government Agency

December 9, 2021

Dr. Jamika D. Burge

"Inclusive teams, especially women-led ones, are more productive and effective at achieving results."

Dr. Jamika D. Burge Mansi Gupta

Advancing the Inclusion of Womxn in Research Practices

September 15, 2022

Jake Burghardt

"Turning every study plan into a kind of report with new and existing insights is a mindset shift that can build a durable asset."

Jake Burghardt

Finding More Inroads into Research Impact

February 20, 2026