Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Kristin Wisnewski
Measuring What Matters
2019 • DesignOps Summit 2019
Gold
Daniel J. Rosenberg
Digital Medicine Design
2019 • Enterprise Community
Frances Yllana
Theme 2 Intro
2024 • DesignOps Summit 2024
Gold
Changying (Z) Zheng
From prototype to production: Vibe coding design for real engineering systems
2026 • Designing with AI 2026
Gold
Erin Weigel
Testing and Experimentation Tools
2026 • Advancing Research 2026
Gold
Jemma Ahmed
Theme 2 Intro
2024 • Enterprise Experience 2020
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold
John Calhoun
Bring your DesignOps Story to Life! The Definitive DesignOps Book Jam
2023 • DesignOps Summit 2023
Gold
Candace Myers
Standardizing Design at Scale
2022 • DesignOps Summit 2022
Gold
Jen Cardello
Standardizing Product Merits for Leaders, Designers, and Everyone
2018 • Enterprise Experience 2018
Gold
Paul Bakaus
Killing the handoff: Iterating design live in the browser on real production code
2026 • Designing with AI 2026
Gold
Uday Gajendar
Theme Four Intro
2023 • Enterprise UX 2023
Gold
Rachael Dietkus, LCSW
AI: Passionate defenses and reasoned critique [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Peter Van Dijck
Hands-on AI #2: Understanding evals: LLM as a Judge
2025 • Rosenfeld Community
Louis Rosenfeld
Becoming a Civic Designer: Making the Move from Private to Public Sector
2022 • Civic Design 2022
Gold
Elana Chapman
Getting started with accessibility research
2025 • Rosenfeld Community

More Videos

Sheri Byrne-Haber

"94% of companies exclude disability from their diversity and inclusion initiatives, a major missed opportunity to advance accessibility."

Sheri Byrne-Haber

The Importance of Accessible Design Systems

January 8, 2024

Nathan Shedroff

"The accounting system only showed Instagram’s value at $86 million; the rest was goodwill representing intangible value."

Nathan Shedroff

Redefining Value: Bridging the Innovation Culture Divide

May 14, 2015

Onur Kocan

"Prost's master plan imposed a heavy interventionist burden on the historical structure of the city."

Onur Kocan Ayhan Ensici

Understanding the Strategy for Civic Design in a Complex City: Istanbul

November 16, 2022

Matt Duignan

"You have to push against 200,000 years of human evolution to get researchers to create reusable knowledge."

Matt Duignan

Atomizing Research: Trend or Trap

March 30, 2020

Alba Villamil

"The word 'requires' implies that whiteness is the default unless otherwise specified."

Alba Villamil

Stereotyped by Design: Pitfalls in Cross-Cultural User Research

March 30, 2020

Llewyn Paine

"DeepFakes can look too realistic and might introduce new privacy risks if donor faces come from real people."

Llewyn Paine

[Demo] Deploying AI doppelgangers to de-identify user research recordings

June 5, 2024

Landon Barnes

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Bria Alexander

"Respect and kindness are mandatory; please read our code of conduct to ensure a positive community experience."

Bria Alexander

Opening Remarks

September 9, 2022

Marc Fonteijn

"Senior roles like VP require negotiating your salary almost by default."

Marc Fonteijn

First Insights from the 2025 Service Design Salary(+) Report

December 4, 2024