Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
What co-speech gestures reveal about users’ thinking during interviews
Summary
Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.
Key Insights
-
•
Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.
-
•
Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.
-
•
Users commonly hold multiple, context-dependent mental models rather than a single static one.
-
•
In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.
-
•
Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.
-
•
Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.
-
•
Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.
-
•
Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.
-
•
Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.
-
•
Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.
Notable Quotes
"Gestures are a window to the mind."
"Gesture and speech form an integrated system; they reinforce one another."
"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."
"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."
"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."
"When gestures and speech mismatch, it often means the speaker is still refining their mental model."
"Without looking at the gesture, we would have come to a less complete mental model."
"Users have more than one mental model; they can be constructed on the fly depending on context."
"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."
"The more you gesture, the more your interviewee will gesture."
Or choose a question:
More Videos
"Reducing onboarding from two weeks to three and a half days lowers overwhelm and improves content retention."
Kate SternScaling Learning for the Future
September 9, 2022
"Only one in 10 companies have quantifiable measurement for their design leaders."
Kristin Skinner Kamdyn Moore8 Types of Measures in Design Operations
April 16, 2020
"It’s not just about doing the craft of design but also figuring out how design fits into the organization."
Francesca Barrientos, PhDYou Need Your Own Definition of Design Maturity
June 8, 2022
"Many leaders didn’t know how to engage with research beyond usability and experimentation."
Mac SmithMeasuring Up: Using Product Research for Organizational Impact
March 12, 2021
"There’s never been a better time for designers to get their hands on tools that seemed far away before."
Shipra Kayan Robert Kortenoeven Eileen TangEmerging principles for using AI in Design: What the product design team at Miro has learned from deeply integrating AI in their workflow
June 11, 2025
"It can be hard for designers to immediately understand or advocate for the real value of design at deeper technical layers."
Stefanie OwensOptimizing for Outcomes: Transformation Design in Systems at Scale
December 4, 2024
"Imagine going to the hospital for an emergency and the doctor doesn’t believe you’re in pain because AI says so."
Dr. Jamika D. BurgeBroad Strokes: Connecting Design, Research, and AI to the World Around Us
June 7, 2023
"Change is fundamentally a design problem and therefore change can be designed."
Maria GiudiceBecoming a Changemaker by Leading with Design
March 29, 2023
"Most users want a quick resolution, so many calls start as problem-solving and then transition to open-ended research conversations."
Prayag Narula Abhinav KrishnaDialing for Research: How to Reach the Unreachable
March 10, 2022
Latest Books All books
Dig deeper with the Rosenbot
How can designers use AI to create entirely new interaction paradigms instead of just improving productivity?
What challenges arise from emerging AI tools that blur the roles of PM, design, and engineering?
How can design guidelines and brand semantics be encoded to help AI and engineering teams work together?