Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Josh Clark
Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)
2025 • Rosenfeld Community
Jorge Arango
Exploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)
2024 • Rosenfeld Community
Erin Hoffman-John
This Game is Never Done: Design Leadership Techniques from the Video Game World
2017 • DesignOps Summit 2017
Gold
Steve Chaparro
Bringing Into Alignment Brand, Culture and Space
2020 • DesignOps Community
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book?
2024 • Rosenfeld Community
Sofía Delsordo
Public Policy for Jalisco's Designers to Make Design Matter
2021 • Civic Design 2021
Gold
Juhan Sonin
Design Now! The Agenda for Action
2025 • Rosenfeld Community
Jonathan Fairman
Integrating generative AI into enterprise products: A case study from dscout
2024 • Designing with AI 2024
Gold
Mansi Gupta
Drawing from Feminist Practice to Make Inclusive Design Operational
2022 • DesignOps Summit 2022
Gold
Andy Polaine
What is the role of service design in product-led organizations?
2024 • Advancing Service Design 2024
Gold
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform
2019 • Advancing Research Community
Ariel Kennan
Building a Design Culture
2017 • Enterprise Experience 2017
Gold
Maria Rosala
Research Repositories
2026 • Advancing Research 2026
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Joanna Vodopivec
One Research Team for All - Influence Without Authority
2022 • Advancing Research 2022
Gold
Chris Engledowl
A Mixed Method Approach to Validity to Help Build Trust
2023 • QuantQual Interest Group

More Videos

Joerg Beringer

"Product managers appreciate the jobs to be done desired outcome statements because they are not overly detailed but actionable."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

September 10, 2025

Marc Rettig

"We’re building a program here that lasts through 2025 and 2030, so we can take risks on entry level folks."

Marc Rettig Julie Baher Phil Gilbert Nathan Shedroff

Discussion

May 14, 2015

Claire Dhoosche

"Even with all our automation, we always need a human to judge the output."

Claire Dhoosche

Coordinating chaos: Preventing workflow fragmentation when everyone accelerates with AI

June 9, 2026

Phil Gilbert

"IBM Design Thinking is about collaborating around artifacts and frameworks, not prescribing the process teams must follow."

Phil Gilbert

A Consistent Culture of Design

May 14, 2015

Jilanna Wilson

"We share the inconvenience of meeting times across time zones, alternating so no one team always bears it."

Jilanna Wilson

Distributed Design Operations Management

October 23, 2019

Rachel Garb

"UX skills are like secret superpowers you bring to any job, even if it’s a small company or non-tech field."

Rachel Garb

Exit Interview #7: Facing a blank slate in retirement when constraints are all you’ve ever known

August 5, 2026

Sarah Fathallah

"Naming is an exercise in power when what is being named has been historically erased."

Sarah Fathallah

Lessening the Research Burden on Vulnerable Communities

March 30, 2020

John Calhoun

"We see design ops as focusing on people, process, and platform — what we act on."

John Calhoun Rachel Posman

Bring your DesignOps Story to Life! The Definitive DesignOps Book Jam

October 3, 2023

Uday Gajendar

"Breaks equalize stress and can prevent trauma responses from spilling over."

Uday Gajendar Rachael Dietkus, LCSW Dr. Dawn Emerick Dawn E. Shedrick, LCSW

Leading through the long tail of trauma

July 7, 2022