Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Kristin Skinner
Opening Keynote: Org Design for Design Orgs
2017 • DesignOps Summit 2017
Gold
Dr. Jamika D. Burge
Broad Strokes: Connecting Design, Research, and AI to the World Around Us
2023 • Enterprise UX 2023
Gold
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Designing with AI 2025
Gold
Louis Rosenfeld
Day 1 Welcome
2024 • DesignOps Summit 2024
Gold
Luke Roberts
Panel Discussion
2024 • Advancing Service Design 2024
Gold
Vincent Brathwaite
Opener: Past, Present, and Future—Closing the Racial Divide in Design Teams
2020 • DesignOps Summit 2020
Gold
Patrick Commarford
Design Staffing for Impact
2024 • DesignOps Summit 2020
Gold
Bria Alexander
Welcome
2022 • DesignOps Summit 2022
Gold
Peter Merholz
The Trials and Tribulations of Directors of UX
2023 • Enterprise Community
Janaki Kumar
Innovate with Purpose
2018 • Enterprise Experience 2018
Gold
Bryce Benton
[Demo] AI-powered UX enhancement: Aligning GitHub documentation with USWDS at Austin Public Library
2024 • Designing with AI 2024
Gold
Satyam Kantamneni
Do You Have an Experience Vision?
2023 • Enterprise Community
Yolanda Rankin
Black Feminist Epistemology as a Critical Framework for Equitable Design
2021 • Advancing Research 2021
Gold
Lisanne Norman
Why I Left Research
2023 • Advancing Research 2023
Gold
Liza Pemstein
Scaling Research Via an Ops First Model at Clever
2023 • Advancing Research 2023
Gold
Sarah Williams
Verizon_A Framework for CX Transformation
2024 • Design at Scale 2021
Gold

More Videos

Jenny Price

"We needed a modern, scalable way to operationalize transformation across more than 4,500 global employees."

Jenny Price

From Tradition to Transformation: Unlocking Startup Agility in a Legacy Enterprise

September 10, 2025

Mac Smith

"We had a great depth of customer understanding, but our view of partners was two-dimensional; we didn't know their needs or goals well."

Mac Smith

Measuring Up: Using Product Research for Organizational Impact

March 12, 2021

Gabriela Barneva

"Service experiences are only as inclusive as their weakest checkpoint."

Gabriela Barneva

Operationalizing Inclusive Design in Service Design

November 20, 2025

Sam Proulx

"You can create prototypes for screen reader users in Word, Excel, or Google Docs by simulating semantic markup."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021

Dave Hora

"Growth as a researcher means expanding your sphere of influence from self to the entire organization."

Dave Hora

A Research Skills Evolution

March 10, 2021

Dana Chisnell

"You can do evolution or you can do revolution. I prefer evolution where everyone collaborates toward the same goals."

Dana Chisnell

The Sensemaking Business

March 10, 2026

Shipra Kayan

"AI clustering gives a first stab at themes, but you have to move things around yourself."

Shipra Kayan

Make your research synthesis speedy and more collaborative using a canvas

January 24, 2025

Rittika Basu

"Localization of content is essential because language barriers affect older adults globally."

Rittika Basu

Age and Interfaces: Equipping Older Adults with Technological Tools

February 23, 2023

Anna Poznyakov

"Our job is to make sure research conducted by non-researchers is objective, rigorous and unbiased."

Anna Poznyakov Richa Prajapati

Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact

March 12, 2021