Rosenverse

Accessible only to conference ticket holders.

Log in Create account Buy conference recordings

For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.

If you have purchased recording access and cannot see the video, please contact support.

Don’t call it AI: Turn words into numbers with quantitative ethnography
Conference ticket
Wednesday, March 11, 2026 • Advancing Research 2026
Share the love for this talk
Don’t call it AI: Turn words into numbers with quantitative ethnography
Speakers: Vitorio Miliano
Link:

Summary

Quantitative ethnography is the niche subfield you’ve never heard of, but it’s one you’ve been increasingly pressured to practice in over the past couple of years. It’s the math that turns words into numbers underlying generative AI, and LLMs have been getting in between you and a radically new approach to working with verbatims, transcripts, and other texts. Business stakeholders are always pushing for greater efficiency, faster turnarounds. Qualitative researchers are always looking for more contact with users, and greater engagement with findings and reporting. Quantitative ethnography (and epistemic network analysis) offers a compromise: by trading structure and semantics for human sensemaking in the analysis part of research, perhaps both groups can get what they want. I’ve had the opportunity to conduct quantitative ethnographic analyses in enterprise studies involving dozens of products, and impacting hundreds of thousands of end-users. Stakeholders were willing to accept a different kind of analysis, and engage more deeply with the process, in exchange for quicker answers. In this talk, I’ll share how quantitative ethnography differs from qualitative ethnography, the tradeoffs you’ll have to make, and the kinds of results you can expect. This isn’t a tools talk, but you won’t need to do any math, either. I’ll close with a look into the near future, one where you can talk with as many users as will take your call with effectively zero additional analysis work; where you can have the analysis running live during your session, and have the user participate in the sensemaking process on-the-fly; and the dream of every product manager, one where stakeholders can have dashboards of evidence updated live as users talk.

Key Insights

  • Quantitative ethnography unifies qualitative ethnographic methods with quantitative statistical validation, avoiding typical mixed-methods back-and-forth.

  • Formalizing coding rules in a detailed code book is essential to scale qualitative insights and enable automation.

  • Defining mechanistic signifiers, such as keywords or phrase rules, is necessary to automate qualitative coding effectively.

  • Intra-sample statistical analysis uses each coded line as a data point rather than each respondent, enabling meaningful stats from small sample sizes.

  • Partnering with data scientists is critical because quantitative ethnography requires specialized, adjusted statistical methods that differ from conventional ones.

  • Researchers must regularly validate coding accuracy and statistical assumptions over time, a process called closing the interpretive loop.

  • Quantitative ethnography can scale from a handful of interviews to thousands of verbatim responses, maintaining rigor at all scales.

  • Epistemic network analysis helps identify and quantify relationships between qualitative codes within the text data.

  • Large language models can automate parts of quantitative ethnography but require sacrificing some control over code definitions and initial synthesis.

  • Quantitative ethnography opens the possibility for near-real-time insights by automating coding and saturation metrics during ongoing data collection.

Notable Quotes

"Business stakeholders push researchers for faster turnarounds and numbers, often favoring surveys over deep interviews."

"Quantitative ethnography isn’t mixed methods; it’s a unified method using both qualitative theory and quantitative validation."

"If you can’t come up with a rule for something, you can’t code it."

"Each coded line is a data point, which enables statistical power even with small numbers of respondents."

"Partner with data scientists to pick and adjust statistical tests because quantitative ethnography requires new assumptions."

"Closing the interpretive loop means regularly checking that your coding and stats hold up as new data arrives."

"Epistemic network analysis reveals meaningful connections between codes, suggesting but not proving why ideas cluster."

"Large language models cluster text using semantic relationships rather than shared vocabulary like traditional QDA."

"Using generative AI math lets you skip stats, but you lose control over what codes start your synthesis."

"If rules and stats update in real time, you could know when saturation is reached as data streams in."

Ask the Rosenbot
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Deanna Smith
Leading Change with Confidence: Strategies for Optimizing Your Process
2024 • DesignOps Summit 2024
Gold
Jon Fukuda
Theme One Intro
2022 • DesignOps Summit 2022
Gold
William Newton
How to Lead With Data, and Without Data
2023 • Enterprise UX 2023
Gold
Erin Weigel
Real-world lessons to improve your conversion rates
2024 • Rosenfeld Community
JD Buckley
COMMUNICATE: Discussion
2018 • Enterprise Experience 2018
Gold
Nidhi Singh Rathore
Embracing participation to unlock deeper truths in commercial research
2025 • Advancing Research 2025
Gold
Michal Anne Rogondino
Saving Outer Space: The First UX Design System for Our Nation’s Satellites
2024 • Enterprise Experience 2020
Gold
Liwei Dai
The Heart and Brain of the AI Research
2020 • Advancing Research 2020
Gold
Bria Alexander
Theme Two Intro
2023 • DesignOps Summit 2023
Gold
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Rachel Posman
A Closer Look at Team Ops and Product Ops (Two Sides of the DesignOps Coin)
2020 • DesignOps Community
Janelle Estes
UX Research Trends
2021 • Advancing Research Community
Christian Crumlish
The Pygmalion Effect: In Which a Vibe Coding Experiment Becomes a Million Lines…
2025 • Rosenfeld Community
Mike Oren
Why Pharmaceutical's Research Model Should Replace Design Thinking
2023 • Advancing Research 2023
Gold
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Gold

More Videos

Scott Jensen

"It became less important to keep doing fresh research and more important to grow the collective team knowledge."

Scott Jensen Sarah Delaney Carmen Liu

Short Take #2: UX/Product Lessons from Your Industry Peers

December 6, 2022

Theresa Slate

"Managers need to audit the feedback their team receives, not just accept it at face value."

Theresa Slate Erin Robertson

Why Changing Hearts & Minds Doesn’t Work When Promoting DE&I Efforts, but Checklists Do

October 4, 2023

Steve Portigal

"People in organizations are already, to some extent, out there talking to users."

Steve Portigal

Looking Back…to Look Ahead

March 26, 2024

Kristin Skinner

"Teachers need actionable insights, not just raw data."

Kristin Skinner

Five Years of DesignOps

September 29, 2021

Ned Dwyer

"At Brex, design ops supports a broad range of design disciplines and also partners with engineers and marketers."

Ned Dwyer Emily Stewart James Wallis

The Intersection of Design and ResearchOps

September 24, 2024

Nicole Bergstrom

"Nothing about us without us—people with disabilities must be involved in design and AI development."

Nicole Bergstrom Anna Cook Kate Kalcevich Saara Kamppari-Miller

AccessibilityOps: Moving beyond “nice to have”

September 19, 2024

Christian Rohrer

"Field studies are the most powerful qualitative method available — nothing else compares for real-world insight."

Christian Rohrer

Insight Types That Influence Enterprise Decision Makers

May 13, 2015

Bria Alexander

"Sponsor sessions are completely free and offer content just as valuable as the main program sessions."

Bria Alexander

Opening Remarks

November 17, 2022

Cassini Nazir

"We’re not feeling what others feel, we’re feeling what we think they’re feeling in our own way."

Cassini Nazir Meah Lin

The Dangers of Empathy: Toward More Responsible Design Research

March 27, 2023