Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Ask the Rosenbot
Sarah Gallimore
Inspire Progress with Artifacts from the Future
2022 • Civic Design 2022
Gold
Aras Bilgen
Who does the math: A designer’s journey in building an AI-based tutoring app
2025 • Designing with AI 2025
Gold
Chui Chui Tan
Global insights: Embracing international and intercultural research for innovation
2025 • Advancing Research 2025
Gold
Uday Gajendar
The Wicked Craft of Enterprise UX
2015 • Enterprise UX 2015
Gold
Mansi Gupta
Women-Centric Research: What, Why, How
2023 • Advancing Research 2023
Gold
Jemma Ahmed
Theme Three Intro
2023 • Advancing Research 2023
Gold
Kristin Skinner
Opening Keynote: Org Design for Design Orgs
2017 • DesignOps Summit 2017
Gold
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform
2019 • Advancing Research Community
Adam Cutler
People + Places + Practices = Outcomes
2016 • Enterprise UX 2016
Gold
Uday Gajendar
Day 2 Welcome
2024 • Designing with AI 2024
Gold
Kate Towsey
ResearchOps AMA with Kate Towsey & Jake Burghardt
2025 • Advancing Research Community
Sam Proulx
Everything You Ever Wanted to Know About Screen Readers
2021 • Design at Scale 2021
Gold
Joel Branch
Humanizing AI: Filling the Gaps with Multi-faceted Research
2021 • Advancing Research 2021
Gold
Steve Chaparro
Bringing Into Alignment Brand, Culture and Space
2020 • DesignOps Community
Stefanie Owens
Optimizing for Outcomes: Transformation Design in Systems at Scale
2024 • Advancing Service Design 2024
Gold
Rachael Dietkus, LCSW
Everything You Need to Know about the Civic Design 2022 Call for Presentations
2022 • Civic Design Community

More Videos

Michelle Chin

"A backlog helps pace and define the design ops practice, managing what’s in scope and what’s aspirational."

Michelle Chin

The DesignOps Starter Kit

September 29, 2021

Louis Rosenfeld

"We are still designers, we’re systems designers, we’re organizational designers, even if we’re not making things."

Louis Rosenfeld Jose Coronado Rachel Posman Guneet Singh Crystal Yan

The Bigger Picture: A Panel Discussion

October 23, 2019

Kristin Skinner

"Non-financial, non-data measurements can reveal much more about behaviors than traditional metrics."

Kristin Skinner Kamdyn Moore

Group Activity: A Deep Dive Into Value and Outcomes

October 23, 2019

Josh Clark

"This isn’t about Skynet; it’s about simple signals of awareness that adapt to immediate context."

Josh Clark Veronika Kindred

Sentient Scenes and Radically Adaptive Experiences

June 11, 2025

JP Allen

"Our subway lines are tool functions, and stations are where single-function tools live."

JP Allen Holly Holden

Navigating the UX Tools Landscape

October 1, 2021

Dr. Jamika D. Burge

"Organizational change management requires as much care and effort as executing good research."

Dr. Jamika D. Burge Robert Fabricant Peter Merholz Noam Segal Teresa Torres

A Genuine Conversation about the Future of UX Research

March 20, 2024

Dan Willis

"Know-it-alls claim to know what to do and how users behave, but if they knew the answers, why bring in research?"

Dan Willis

Enterprise Storytelling Sessions

June 8, 2016

"I feel like I play more of a chief of staff type role than just head of design operations."

Panel Discussion: Communicating the Value of DesignOps

November 7, 2018

Husani Oakley

"AI in design tooling can be amazing, terrible, empowering, and creepy all at once."

Husani Oakley

Theme Three Intro

June 6, 2023