Investigating qualitative depth of AI-moderated interviews
Summary
Faced with the need to understand our customers’ decision-making (those who purchased our platform, and those who didn’t), I initially relied on surveys. However, I found they lacked the nuance that real conversations reveal. By introducing AI moderation, I created a way for participants to engage in adaptive, conversational interviews that went beyond the limits of static forms.
I’ll walk through how I set up these sessions, what prompts worked (and didn’t), and how I analyzed the results. I’ll also share how I’ve used other AI tools like ChatGPT and Perplexity to assist with synthesis and bias-checking, creating a workflow that both expands my analytical reach and strengthens the rigor of my findings.
As AI tools continue to enter the researcher’s toolkit, this case study illustrates how we can thoughtfully integrate them to expand—not erode—the human depth of qualitative work. It offers a model for how lean teams can maintain research quality while navigating the realities of limited time, budget, and capacity.
This talk will explore the emerging space between automation and augmentation, finding opportunity for depth when time and resources are tight.”
Key Insights
-
•
AI moderation enables more qualitative depth than surveys due to participants speaking freely rather than typing.
-
•
AI moderation platforms require detailed and explicit research plans to guide effective questioning.
-
•
AI moderation often struggles with natural conversational flow and can produce awkward pauses or interruptions.
-
•
Follow-up questions from AI moderators can be vague, repetitive, or miss opportunities to dive deeper.
-
•
AI moderation is best suited for validation or deepening understanding of known problem spaces, not foundational discovery.
-
•
Participants may prefer AI moderation over surveys due to ease and flexibility, supporting higher and quicker recruitment.
-
•
AI moderation platforms currently do not replace human analysis or provide strong quantitative outputs.
-
•
High emotion or sensitive topics may work better with AI moderation if participants prefer anonymity, but human interviews excel at rapport.
-
•
Researchers need to invest time troubleshooting and iterating prompts in AI moderated studies.
-
•
AI moderated studies allow teams of one to scale qualitative research when live interviews are infeasible.
Notable Quotes
"Having one-on-calls with people is one of my favorite parts of conducting research, and it felt like this was being taken away from me just for the sake of saving business resources."
"AI moderation, in short, is a large language model that runs a moderated interview session without a human researcher present."
"I was interested in thinking of this method as a survey plus that you can use to get survey-like data but with more qualitative depth."
"When I gave the model my list of interview questions, it actually made for a really bizarre participant experience because the model just went down the line of the different questions."
"People were more likely to explain their thought process and just more context around the particular situation of their org."
"There were moments like long kind of awkward pauses where the model was processing, and sometimes it actually spoke over participants."
"Follow-up questions missed opportunities to probe deeper or were confusing, and participants had to repeat themselves."
"If I really just needed only quantitative data from a survey, then I would not go the AI moderated route."
"AI moderation did not make the research easier. It made it possible under real world constraints."
"From setup to analysis, AI moderation still requires a good understanding of what quality research looks like and how to interpret qualitative data."
Or choose a question:
More Videos
"Many customers struggle to get anyone to listen to actual insights because of the volume and complexity of raw video data."
Andy Barraclough Betsy NelsonFrom Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
September 23, 2024
"Overlaying cultural dialogue on user journey maps uncovers insightful tensions and opportunities."
Neil BarrieWidening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
March 25, 2024
"When you hit the bottom, that’s where creativity flourishes and it’s time to iterate, evolve, and redesign."
Maria GiudiceBecoming a Changemaker by Leading with Design
March 29, 2023
"They wouldn’t tell us what the requirements were; it was our job to figure it out."
Joshua GravesWe Need To Talk: Managing Ludicrous Requests at Work (Part 3 of 3)
May 12, 2025
"Researchers can overcome reluctance to include others by training them closely and collaborating on synthesis to maintain data quality."
Dr Chloe SharpUsing Evidence and Collaboration for Setting and Defending Priorities
November 29, 2023
"I like to engage my stakeholders as co-creators and co-discoverers so their ideas and concerns get reflected in recommendations."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."
Kristin WisnewskiMeasuring What Matters
October 23, 2019
"Storytelling has the power to disrupt hegemonic narratives and open space for people to be authors of their own stories."
Bilan HashiThe Tension Between Story Collecting and Story Telling in Research
March 10, 2021
"You have to become seen as an indispensable resource to your business partners by constantly asking what value you bring."
Brianna SylverLead With Purpose
March 31, 2020
Latest Books All books
Dig deeper with the Rosenbot
What lessons from the 1990s web standards movement apply to AI interface standardization today?
How can organizations prevent design system knowledge from being an undocumented API surface reliant on specific individuals?
What tools enable local, privacy-focused integration of smart home energy data without relying on cloud services?