Rosenverse

Accessible only to conference ticket holders.

Log in Create account Buy conference recordings

For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.

If you have purchased recording access and cannot see the video, please contact support.

Don’t call it AI: Turn words into numbers with quantitative ethnography
Conference ticket
Wednesday, March 11, 2026 • Advancing Research 2026
Share the love for this talk
Don’t call it AI: Turn words into numbers with quantitative ethnography
Speakers: Vitorio Miliano
Link:

Summary

Quantitative ethnography is the niche subfield you’ve never heard of, but it’s one you’ve been increasingly pressured to practice in over the past couple of years. It’s the math that turns words into numbers underlying generative AI, and LLMs have been getting in between you and a radically new approach to working with verbatims, transcripts, and other texts. Business stakeholders are always pushing for greater efficiency, faster turnarounds. Qualitative researchers are always looking for more contact with users, and greater engagement with findings and reporting. Quantitative ethnography (and epistemic network analysis) offers a compromise: by trading structure and semantics for human sensemaking in the analysis part of research, perhaps both groups can get what they want. I’ve had the opportunity to conduct quantitative ethnographic analyses in enterprise studies involving dozens of products, and impacting hundreds of thousands of end-users. Stakeholders were willing to accept a different kind of analysis, and engage more deeply with the process, in exchange for quicker answers. In this talk, I’ll share how quantitative ethnography differs from qualitative ethnography, the tradeoffs you’ll have to make, and the kinds of results you can expect. This isn’t a tools talk, but you won’t need to do any math, either. I’ll close with a look into the near future, one where you can talk with as many users as will take your call with effectively zero additional analysis work; where you can have the analysis running live during your session, and have the user participate in the sensemaking process on-the-fly; and the dream of every product manager, one where stakeholders can have dashboards of evidence updated live as users talk.

Key Insights

  • Quantitative ethnography unifies qualitative ethnographic methods with quantitative statistical validation, avoiding typical mixed-methods back-and-forth.

  • Formalizing coding rules in a detailed code book is essential to scale qualitative insights and enable automation.

  • Defining mechanistic signifiers, such as keywords or phrase rules, is necessary to automate qualitative coding effectively.

  • Intra-sample statistical analysis uses each coded line as a data point rather than each respondent, enabling meaningful stats from small sample sizes.

  • Partnering with data scientists is critical because quantitative ethnography requires specialized, adjusted statistical methods that differ from conventional ones.

  • Researchers must regularly validate coding accuracy and statistical assumptions over time, a process called closing the interpretive loop.

  • Quantitative ethnography can scale from a handful of interviews to thousands of verbatim responses, maintaining rigor at all scales.

  • Epistemic network analysis helps identify and quantify relationships between qualitative codes within the text data.

  • Large language models can automate parts of quantitative ethnography but require sacrificing some control over code definitions and initial synthesis.

  • Quantitative ethnography opens the possibility for near-real-time insights by automating coding and saturation metrics during ongoing data collection.

Notable Quotes

"Business stakeholders push researchers for faster turnarounds and numbers, often favoring surveys over deep interviews."

"Quantitative ethnography isn’t mixed methods; it’s a unified method using both qualitative theory and quantitative validation."

"If you can’t come up with a rule for something, you can’t code it."

"Each coded line is a data point, which enables statistical power even with small numbers of respondents."

"Partner with data scientists to pick and adjust statistical tests because quantitative ethnography requires new assumptions."

"Closing the interpretive loop means regularly checking that your coding and stats hold up as new data arrives."

"Epistemic network analysis reveals meaningful connections between codes, suggesting but not proving why ideas cluster."

"Large language models cluster text using semantic relationships rather than shared vocabulary like traditional QDA."

"Using generative AI math lets you skip stats, but you lose control over what codes start your synthesis."

"If rules and stats update in real time, you could know when saturation is reached as data streams in."

Ask the Rosenbot
Mackenzie Cockram
Integrating Qualitative and Quantitative Research from Discovery to Live
2022 • QuantQual Interest Group
Molly Fargotstein
Multipurpose Communication & UX Research Marketing
2019 • DesignOps Community
Smitha Papolu
Theme 3 Discussion
2024 • Enterprise Experience 2020
Gold
Holly Cole
Panel Discussion: Growing People and Teams
2018 • DesignOps Summit 2018
Gold
Doug Powell
Closing Keynote: Design at Scale
2018 • DesignOps Summit 2018
Gold
Marjorie Stainback
Transforming Strategic Research Capacity through Democratization
2019 • DesignOps Summit 2019
Gold
Eniola Oluwole
Lessons From the DesignOps Journey of the World's Largest Travel Site
2019 • DesignOps Summit 2019
Gold
Sam Proulx
Mobile Accessibility and You
2022 • Design at Scale 2022
Gold
Shelby Switzer
Making Space for Community Knowledge-sharing in a Distributed World
2021 • Civic Design 2021
Gold
Jack Moffett
SAFe or Sorry?
2019 • Enterprise Community
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Advancing Research 2022
Gold
Victor M. Gonzalez
Practicing Learners and Learning Practitioners
2021 • Advancing Research 2021
Gold
Alla Weinberg
People Are Sick of Change: Psychological Safety is the Cure
2023 • DesignOps Community
Jessamyn Edwards
Surviving Your UX Career in Enterprise Design
2021 • Enterprise Community
Peter Merholz
The Trials and Tribulations of Directors of UX
2023 • Enterprise Community
Meaghan Waters
Lack of Product Thinking will Doom Your Legacy Modernization
2021 • Design at Scale 2021
Gold

More Videos

Louis Rosenfeld

"There is no need to take notes over the next few days. We have dedicated sketch notes and session resources."

Louis Rosenfeld Bria Alexander

Opening Remarks

March 27, 2023

Gregg Bernstein

"Relentless oversharing of research in Slack channels builds awareness and invites collaboration."

Gregg Bernstein

Opportunistic Research with Gregg Bernstein

July 11, 2019

Emily Eagle

"Truly listening can push us beyond empathy to respect and reflection."

Emily Eagle

Can't Rewind: Radio and Retail

June 3, 2019

Alison Rand

"COVID forced all of us into vulnerability, letting us see each other as humans beyond our work roles."

Alison Rand Sarah Brooks

Scaling Impact with Service Design

March 25, 2021

Dan Saffer

"Most AI fails because projects require near perfect accuracy, which AI can't reliably deliver."

Dan Saffer

Why AI projects fail (and what we can do about it)

May 14, 2025

Suzan Bednarz

"We don't assume companies intentionally fail certain audiences; it’s often about awareness and lacking champions."

Suzan Bednarz Hilary Sunderland

AccessibilityOps for All

January 8, 2024

Peter Levin

"If you don’t know where you are going, try to make the organization more flexible and open to possibilities."

Peter Levin

Solve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility

March 25, 2024

Megan Blocker

"Maturity is not just knowing what to do, but also knowing what not to do and being smart and grown up about saying no."

Megan Blocker Lada Gorlenko Fatimah Richmond Molly Stevens

What UX research maturity looks like and how we get there [Advancing Research Community Workshop Series]

November 9, 2023

Theresa Marwah

"Building respect is the foundation to get insights that truly help us deliver to our customers' needs."

Theresa Marwah

How Atlassian is Operationalizing Respect in Research

February 27, 2020