Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Cheryl Platz
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
2024 • Rosenfeld Community
Sharon Banh
Reimagining research: What does the field need to grow? [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
John Mortimer
Panel Discussion
2024 • Advancing Service Design 2024
Gold
Joshua Graves
We Need To Talk: Addressing Unmet Expectations (Part 2 of 3)
2025 • Rosenfeld Community
Bria Alexander
Reflect and Chart Forward
2021 • Civic Design 2021
Gold
Uday Gajendar
Leading through the long tail of trauma
2022 • Advancing Research Community
Sheryl Cababa
Integrating Systems Thinking Into Your Practice as a Designer
2025 • Rosenfeld Community
Peter Merholz
The 2025 State of UX/Design Organizational Health
2025 • Rosenfeld Community
Jaskiran Kang
Why Community is Key to Professionalizing Design
2022 • Civic Design Community
Chris Govias
Perspectives on Civic Design
2021 • Civic Design Community
Lily Aduana
5 Reasons to Bring Your Recruiting in-House (and How To Do It)
2021 • Advancing Research 2021
Gold
Scott Stephens
The Next Generation in DesignOps Toolsets
2022 • DesignOps Community
Jose Coronado
People First - Design at JP Morgan
2021 • Design at Scale 2021
Gold
Phil Hesketh
Designing Accessible Research Workflows
2021 • DesignOps Summit 2021
Gold
Greg Petroff
Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
2025 • Rosenfeld Community
Alla Weinberg
Workers Are Sick of Change: The Cure is Psychological Safety
2023 • Enterprise UX 2023
Gold

More Videos

Mitchell Bernstein

"Every decision in a design system is like a puzzle piece; together they form a bigger picture that evolves over time."

Mitchell Bernstein

Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape

September 29, 2021

Catt Small

"Keeping your craft sharp, learning new tools like auto layout in Figma, and challenging yourself help maintain relevance."

Catt Small Micah Bennett Brian Carr Jessica Harllee

What's Next for ICs: Exploring Staff and Principal Designer Roles

February 22, 2024

Victor Udoewa

"There are many ways to compensate people, but it is not your choice—it is theirs."

Victor Udoewa

Radical Participatory Research: Decolonizing Participatory Processes

March 9, 2022

Dane DeSutter

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

Dane DeSutter Stephanie Scopelitis

What co-speech gestures reveal about users’ thinking during interviews

June 30, 2023

Jen Cardello

"The minimum bar for collaborative insight generation is the ability to speak the same language across disciplines."

Jen Cardello Dr. Shadi Janansefat Alex Wright

Curating insight: Strategies for integrating knowledge across research functions

March 11, 2025

Brennan Hartich

"Allocating a DPM to a team that didn’t want one caused more confusion and extra headcount."

Brennan Hartich

Communicating and Establishing DesignOps as a New Function

November 7, 2018

Luke Wroblewski

"The evolution in AI UIs is ongoing and rapid, and we’re still figuring out what the final form of AI-native applications looks like."

Luke Wroblewski

Designing AI Applications

March 5, 2026

Taiye Akin-Akinyosoye

"Moderation is a performance; the best focus groups feel like natural conversations, not rigid Q&A."

Taiye Akin-Akinyosoye

Amplifying voices and enhancing user research through group interviews

March 12, 2025

Ricardo Martins

"Conjoint analysis reveals how consumers make choices by examining trade-offs they are willing to make."

Ricardo Martins

Unlocking the power of advanced quantitative methods

March 12, 2025