Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
David Eisner
From prototype to production: Vibe coding design for real engineering systems
2026 • Designing with AI 2026
Conference
Chelsey Glasson
Exit Interview #3: Same as It Ever Was: What Leaving Tech Taught Me About Change
2025 • Rosenfeld Community
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
Opening Remarks
2022 • Civic Design 2022
Gold
Dave Hora
Research in the Face of Complexity: New Sensibility for New Situations
2025 • Rosenfeld Community
Jack Behar
How to Build Prototypes that Behave like an End-Product
2022 • Design in Product 2022
Gold
Sahibzada Mayed
Cultivating Design Ecologies of Care, Community, and Collaboration
2023 • DesignOps Summit 2023
Gold
Ari Zelmanow
Dark Side of Democratization
2023 • Advancing Research Community
Patrizia Bertini
Pushing DesignOps’ Influence into New Global Markets
2022 • DesignOps Summit 2022
Gold
Samuel Martin
Co-Design vs Faux-Design: Navigating the Complexities of Sharing Power in Co-Design
2026 • Rosenfeld Community
Sam Ladner
Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX
2016 • Enterprise UX 2016
Gold
Alba Villamil
Stereotyped by Design: Pitfalls in Cross-Cultural User Research
2020 • Advancing Research 2020
Gold
Jilanna Wilson
Distributed Design Operations Management
2019 • DesignOps Summit 2019
Gold
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Peter Merholz
Customer-Centered Design Organizations
2017 • Enterprise Experience 2017
Gold
Catt Small
Craft a Vision that Actually Gets Shipped
2026 • Rosenfeld Community

More Videos

Dane DeSutter

"Without looking at the gesture, we would have come to a less complete mental model."

Dane DeSutter Stephanie Scopelitis

What co-speech gestures reveal about users’ thinking during interviews

June 30, 2023

Anat Fintzi

"We consolidated over ten tools into one supply chain management system with a user-led team of 4 UXers, 2 product managers, and 6 engineers."

Anat Fintzi Rachel Minnicks

Delivering at Scale: Making Traction with Resistant Partners

June 9, 2022

Eduardo Ortiz

"Trauma among participants and practitioners is an industry problem we can no longer ignore."

Eduardo Ortiz

Theme 3 Intro

March 13, 2025

Rachael Dietkus, LCSW

"In design, it's the BYOE model—bring your own ethics."

Rachael Dietkus, LCSW

The power to heal and harm

March 13, 2025

Sahibzada Mayed

"Empathy is really just the floor, not the ceiling; it cannot replace someone's actual lived experience."

Sahibzada Mayed

The Politics of Radical Research: A Manifesto

March 27, 2023

Dave Hora

"Democratization of research is interesting but not a one-size-fits-all; it needs quick assessment of what a team truly needs."

Dave Hora

Research in the Face of Complexity: New Sensibility for New Situations

August 27, 2025

Rikki Teeters

"AI writing code at parity with humans is a huge achievement, and it’s expected to surpass humans within a year."

Rikki Teeters

Concept to code: Transforming ideas into functional products with Kiro

June 10, 2026

Sarah Brooks

"I was largely wrong about how we thought technology and design would just implement policy."

Sarah Brooks Jennifer Pahlka

Fireside chat with Sarah Brooks and Jen Pahlka

October 21, 2021

Bria Alexander

"These free models flounder, but high-end enterprise models are more accurate and context-aware - Brittany."

Bria Alexander Brittany Hobbs Christopher Noessel

Day 1 Panel: Up to the Minute: The latest in AI’s impact on UX

June 10, 2025