Rosenverse
Don’t call it AI: Turn words into numbers with quantitative ethnography

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Don’t call it AI: Turn words into numbers with quantitative ethnography

Gold
Wednesday, March 11, 2026 • Advancing Research 2026
Share the love for this talk
Don’t call it AI: Turn words into numbers with quantitative ethnography
Speakers: Vitorio Miliano
Link:

Summary

Quantitative ethnography is the niche subfield you’ve never heard of, but it’s one you’ve been increasingly pressured to practice in over the past couple of years. It’s the math that turns words into numbers underlying generative AI, and LLMs have been getting in between you and a radically new approach to working with verbatims, transcripts, and other texts. Business stakeholders are always pushing for greater efficiency, faster turnarounds. Qualitative researchers are always looking for more contact with users, and greater engagement with findings and reporting. Quantitative ethnography (and epistemic network analysis) offers a compromise: by trading structure and semantics for human sensemaking in the analysis part of research, perhaps both groups can get what they want. I’ve had the opportunity to conduct quantitative ethnographic analyses in enterprise studies involving dozens of products, and impacting hundreds of thousands of end-users. Stakeholders were willing to accept a different kind of analysis, and engage more deeply with the process, in exchange for quicker answers. In this talk, I’ll share how quantitative ethnography differs from qualitative ethnography, the tradeoffs you’ll have to make, and the kinds of results you can expect. This isn’t a tools talk, but you won’t need to do any math, either. I’ll close with a look into the near future, one where you can talk with as many users as will take your call with effectively zero additional analysis work; where you can have the analysis running live during your session, and have the user participate in the sensemaking process on-the-fly; and the dream of every product manager, one where stakeholders can have dashboards of evidence updated live as users talk.

Key Insights

  • Quantitative ethnography unifies qualitative ethnographic methods with quantitative statistical validation, avoiding typical mixed-methods back-and-forth.

  • Formalizing coding rules in a detailed code book is essential to scale qualitative insights and enable automation.

  • Defining mechanistic signifiers, such as keywords or phrase rules, is necessary to automate qualitative coding effectively.

  • Intra-sample statistical analysis uses each coded line as a data point rather than each respondent, enabling meaningful stats from small sample sizes.

  • Partnering with data scientists is critical because quantitative ethnography requires specialized, adjusted statistical methods that differ from conventional ones.

  • Researchers must regularly validate coding accuracy and statistical assumptions over time, a process called closing the interpretive loop.

  • Quantitative ethnography can scale from a handful of interviews to thousands of verbatim responses, maintaining rigor at all scales.

  • Epistemic network analysis helps identify and quantify relationships between qualitative codes within the text data.

  • Large language models can automate parts of quantitative ethnography but require sacrificing some control over code definitions and initial synthesis.

  • Quantitative ethnography opens the possibility for near-real-time insights by automating coding and saturation metrics during ongoing data collection.

Notable Quotes

"Business stakeholders push researchers for faster turnarounds and numbers, often favoring surveys over deep interviews."

"Quantitative ethnography isn’t mixed methods; it’s a unified method using both qualitative theory and quantitative validation."

"If you can’t come up with a rule for something, you can’t code it."

"Each coded line is a data point, which enables statistical power even with small numbers of respondents."

"Partner with data scientists to pick and adjust statistical tests because quantitative ethnography requires new assumptions."

"Closing the interpretive loop means regularly checking that your coding and stats hold up as new data arrives."

"Epistemic network analysis reveals meaningful connections between codes, suggesting but not proving why ideas cluster."

"Large language models cluster text using semantic relationships rather than shared vocabulary like traditional QDA."

"Using generative AI math lets you skip stats, but you lose control over what codes start your synthesis."

"If rules and stats update in real time, you could know when saturation is reached as data streams in."

Ask the Rosenbot
Prayag Narula
HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer
2023 • Advancing Research 2023
Gold
Ross Smith
Breaking Barriers with Empathy
2017 • Enterprise Experience 2017
Gold
Louis Rosenfeld
How to use the Rosenbot
2026 • Rosenfeld Community
Jane Reid
Self-care in User Research
2020 • Advancing Research Community
Eduardo Ortiz
Theme 3 Intro
2025 • Advancing Research 2025
Gold
Andrew Webster
Scaling Design Capability: How Involved Should You Be?
2021 • DesignOps Summit 2021
Gold
John Devanney
The Design Management Office
2017 • DesignOps Summit 2017
Gold
Dharani Perera
The mandala of service design: unlocking alignment and action through service design
2025 • Advancing Service Design 2025
Gold
Bria Alexander
Reflect and Chart Forward
2021 • Civic Design 2021
Gold
Maggie Dieringer
Cutting through the Noise
2020 • DesignOps Community
Terry Buckman
Wargaming (An Introduction)
2023 • Enterprise Community
Ned Dwyer
The Future of DesignOps is Tool Consolidation
2024 • DesignOps Summit 2024
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • Civic Design 2021
Gold
Barb Spanton
Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
2022 • Design at Scale 2022
Gold
Kritika Sony
Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design
2026 • Designing with AI 2026
Conference
Alexandra Schmidt
Enterprise UX Playbook
2022 • Enterprise Community

More Videos

Bria Alexander

"Respect and kindness are mandatory; please read our code of conduct to ensure a positive community experience."

Bria Alexander

Opening Remarks

September 9, 2022

Tricia Wang

"Working from home is fun only when you can actually leave your home; during a pandemic, it’s incredibly hard."

Tricia Wang

Spatial Collapse: Designing for Emergent Culture

January 8, 2024

Peter Merholz

"Design impact doesn’t exist separate from the work with others – it’s part of the initiative's success."

Peter Merholz

The Trials and Tribulations of Directors of UX

July 13, 2023

Victor Udoewa

"Research doesn’t have a governing methodology; there is only presence, place, relationship, and practice."

Victor Udoewa

Research in the Pluriverse

March 29, 2023

Rachael Dietkus, LCSW

"Discomfort should not be dismissed; it’s a signpost for necessary change."

Rachael Dietkus, LCSW

The power to heal and harm

March 13, 2025

Marjorie Stainback

"Don’t schedule your pilot during the holidays — our first pilot had one graduate because it went through December."

Marjorie Stainback Kelsey Kingman

Transforming Strategic Research Capacity through Democratization

October 24, 2019

Sahibzada Mayed

"You need to show up as humans first before we show up as employees."

Sahibzada Mayed Lauren Lin

Cultivating Design Ecologies of Care, Community, and Collaboration

October 4, 2023

Paul Bakaus

"Good direction is vocabulary based; you need constraints, references, and a point of view."

Paul Bakaus

Killing the handoff: Iterating design live in the browser on real production code

June 10, 2026

Neema Mahdavi

"Our research data had to be trackable and compliant with GDPR, so we automated metadata extraction."

Neema Mahdavi

Operationalizing DesignOps

November 7, 2018