Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Dan Willis
Enterprise Storytelling Sessions
2019 • Enterprise Experience 2019
Gold
Josh Clark
Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)
2025 • Rosenfeld Community
Theresa Marwah
How Atlassian is Operationalizing Respect in Research
2020 • Advancing Research Community
Liza Pemstein
Scaling Research Via an Ops First Model at Clever
2023 • Advancing Research 2023
Gold
Dan Willis
Enterprise Storytelling Sessions
2018 • Enterprise Experience 2018
Gold
Sharbani Dhar
Breathing Room for Delight
2024 • Enterprise Experience 2020
Gold
Greg Petroff
Everything is About to Change: Software as Material
2016 • Enterprise UX 2016
Gold
Helen Armstrong
Augment the Human. Interrogate the System.
2023 • Enterprise UX 2023
Gold
Kyria Stephens
Power to Heal: Civic Design in the Aftermath of Tragedy
2022 • Civic Design 2022
Gold
David Sternberg
Uncovering the hidden forces shaping user behavior
2025 • Rosenfeld Community
Magdalena Zadara
Zero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late
2022 • Civic Design 2022
Gold
James Chudley
Decarbonising User Journeys: How minimising enables us to do more with less
2025 • Climate UX Interest Group
Abbey Smalley
Scaling UX Past the Size of Your Team
2024 • Enterprise Experience 2020
Gold
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Steve Portigal
Discussion
2015 • Enterprise UX 2015
Gold
Jemma Ahmed
Theme Panel
2025 • Advancing Research 2025
Gold

More Videos

Nicole Aleong

"There’s a lot of potential in UX research to go beyond product and influence strategy and C-suite decision making."

Nicole Aleong Michaela Mora Prayag Narula Brianna Sylver

What UX research can learn from other research practices [Advancing Research Community Workshop Series]

September 14, 2023

Laura Klein

"Lawyers, though nervous, can become some of the most reasonable people you deal with if you explain risks clearly."

Laura Klein

Unique challenges of innovation in enterprises

April 23, 2020

Erin Weigel

"We are the shopkeepers of today; it just looks a little bit different."

Erin Weigel

Real-world lessons to improve your conversion rates

June 26, 2024

Changying (Z) Zheng

"Let’s lead with curiosity, govern with integrity, and courageously transform the landscape of design operations."

Changying (Z) Zheng

Navigating Innovation with Integrity

September 25, 2024

Dagmara Kukawka

"We do research not just to generate insights but to embed empathy across the organization."

Dagmara Kukawka

Tiny team, moonshot impact: Democratizing research across continents

March 10, 2026

Elana Chapman

"If you’re not doing anything about accessibility, you’re discriminating against your future self."

Elana Chapman Li Wen Huang Divyen Sanganee Annabel Weiner

Getting started with accessibility research

February 20, 2025

Iain McMaster

"Business language is for communication. We use it to explain why user centricity matters to broader audiences."

Iain McMaster IHan Cheng

Design and Product: from Frenemy to Harmony

November 29, 2023

Ian Swinson

"Careers as ladders are what HR wants, but careers aren’t linear; they should be a jungle gym."

Ian Swinson

Designing and Driving UX Careers

June 8, 2016

Paula Bach

"In the small enterprise user base scenario, you can get down and dirty to know exactly how people use the software and how habit changes will affect them."

Paula Bach

Improving Legacy Software: How Much Better Does it Have to Be?

March 11, 2022