Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

[Demo] Stress-testing GenAI in user research synthesis
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
[Demo] Stress-testing GenAI in user research synthesis
Speakers: Yulya Besplemennova
Link:

Summary

In this high-pressure scenario, the challenge was to conduct 17 user interviews in three days and synthesize a comprehensive report in just one additional day. I’ll explore how we used AI to streamline the research process, from transcription to synthesis, and how tools like ChatGPT contributed to efficient data processing and insight generation. We’ll reflect on the potential and pitfalls of using AI in accelerated user research, from practical aspects to more philosophical considerations on potential changes to the research process. Takeaways Practical insights into integrating AI with traditional research methodologies to expedite the research process An overview of the effectiveness of AI transcription and synthesis tools in real-world research scenarios Critical examination of AI's role in data processing and how it compares with human analysis Strategic considerations for service designers when employing AI to support rapid user research Reflection on the ethical implications and potential impact on the quality of insights and researcher well-being when relying on AI to speed up research processes

Key Insights

  • Generative AI speeds textual content generation and navigation but needs human guidance for meaningful interpretation.

  • AI struggles to track and assign specific interviewees to user archetypes without explicit human instruction.

  • A semi-automated workflow combining AI brainstorming and human refinement delivers faster synthesis under tight deadlines.

  • ChatGPT excels at brainstorming clustering parameters and generating detailed archetype descriptions when properly guided.

  • AI can retrieve relevant interview quotes but often requires multiple prompt iterations to produce longer, contextually fitting excerpts.

  • Visual generation of archetypes using DALL·E currently yields poor results and often contradicts instructions.

  • Token limits can constrain transcript processing, but newer models like GPT-4 improve handling of larger inputs.

  • AI behaves like an 'alien intern'—powerful but unaware of design research processes and simple context details unless taught.

  • Using AI as a creative partner helps overcome cognitive overload, especially in rapid, solo research scenarios.

  • Less experienced researchers require caution when using AI tools, as expert evaluation is crucial for output quality.

Notable Quotes

"Gene AI can really help to speed up synthesis moments connected to textual content generation and navigation."

"AI behaves like an alien intern because it’s not capable of making sense of some things easy for us."

"I approached the GPT as if it were my human colleague, asking very open questions."

"It couldn’t find specific interviewees for user archetypes, so I had to adapt and guide the process."

"This is a hero moment because it accelerates and simplifies exploration of clustering options."

"Text generation is the biggest superpower of ChatGPT in research synthesis right now."

"Visuals from DALL·E didn’t work well; I asked multiple times to remove cables but they still appeared."

"AI can help manage cognitive overload and speed up work when you have short timeframes and no team."

"You need a lot of annoying handholding with the AI: reminding it of how many people you have or simple things."

"I wouldn’t leave AI tools to juniors because it’s crucial to have your own experience to evaluate outputs."

Luz Bratcher
This Is a Talk for Tired People
2022 • Design at Scale 2022
Gold
Nicole Aleong
What UX research can learn from other research practices [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Liza Pemstein
Scaling Research Via an Ops First Model at Clever
2023 • Advancing Research 2023
Gold
JP Allen
Navigating the UX Tools Landscape
2021 • DesignOps Summit 2021
Gold
Dan Willis
Enterprise Storytelling Sessions
2018 • Enterprise Experience 2018
Gold
Frances Yllana
D.E.A.R.R. Diaries (Discipline, Experience, Architecture, Reflection + Revolution)
2022 • Civic Design 2022
Gold
Dr. Karl Jeffries
The Science of Creativity for DesignOps
2024 • DesignOps Summit 2020
Gold
Patrick Boehler
Fishing for Real Needs: Reimagining Journalism Needs with AI
2025 • Designing with AI 2025
Gold
George Hinchliffe
Delivering Amazing Experiences
2021 • Design at Scale 2021
Gold
Bria Alexander
Opening Remarks
2022 • DesignOps Summit 2022
Gold
Carol Smith
Operationalizing Responsible, Human-Centered AI
2023 • Enterprise UX 2023
Gold
Matteo Gratton
Can Data and Ethics Live Together?
2021 • DesignOps Summit 2021
Gold
Dem Gerolemou
Climate technology fundamentals
2024 • Climate UX Interest Group
Kelly Dern
AI as a Design Partner: How to Get the Most Out of AI Tools to Scale Your Process
2023 • DesignOps Summit 2023
Gold
Leah Buley
Ask Me Anything with Leah Buley and Joe Natoli, co-authors of The User Experience Team of One (2nd edition)
2024 • Rosenfeld Community
Sara Conklin
A UXer’s 12-Month Journey from Climate Concern to Climate Credibility
2025 • Climate UX Interest Group

More Videos

Yoel Sumitro

"We should open our profession to fresh graduates and those without traditional design research backgrounds."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Zariah Cameron

"Stop saying rest is a luxury or a privilege. It is not. It is a human right."

Zariah Cameron

ReDesigning Wellbeing for Equitable Care in the Workplace

September 23, 2024

Doug Powell

"Professional development budgets get cut first in tough times, yet skill building is more important than ever."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Darian Davis

"Taking responsibility for our behaviors starts with an apology and continues with regular feedback and action."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Mike Oren

"The essence of strategy isn’t painting a grand vision of the future, it's deciding what not to do."

Mike Oren

Why Pharmaceutical's Research Model Should Replace Design Thinking

March 28, 2023

Jacqui Frey

"One operations person for every ten designers is when you start to feel the pain of coordination."

Jacqui Frey

Scale is Social Work

March 19, 2020

Kim Holt

"Onboarding feels like somebody's holding your hand, even though you're remote, which makes a huge difference."

Kim Holt Emma Wylds Pearl Koppenhaver Maisee Xiong

A Salesforce Panel Discussion on Values-Driven DesignOps

September 8, 2022

Samuel Proulx

"You own the design system, which gives you the unique ability to integrate that accessibility thinking into all of your components."

Samuel Proulx

From Standards to Innovation: Why Inclusive Design Wins

September 10, 2025

Laine Riley Prokay

"Relationship design is not just what you do but how you do it, focusing on building relationships with customers, employees, and community."

Laine Riley Prokay

How DesignOps can Drive Inclusive Career Ladders for All

September 30, 2021