Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Robert Fabricant
Shifting dynamics: The evolving relationship between researchers, participants, and organizational systems
2025 • Advancing Research 2025
Gold
Laureen Kattan
Centering Patients and Clinicians in a Complex Government Ecosystem
2023 • Design in Product 2023
Gold
Vincent Brathwaite
Opener: Past, Present, and Future—Closing the Racial Divide in Design Teams
2020 • DesignOps Summit 2020
Gold
Ryan Rumsey
Business Influence Without Losing Your Soul
2021 • Enterprise Community
Chris Geison
What is Research Strategy?: A Panel of Research Leaders Discuss this Emergent Question
2021 • Advancing Research Community
Dalia El-Shimy
So You've Got a Seat at the Table. Now What?
2020 • Advancing Research 2020
Gold
Changying (Z) Zheng
From prototype to production: Vibe coding design for real engineering systems
2026 • Designing with AI 2026
Conference
Amy Parness
Scaling Sustainability: Complementary strategies that drive long-term success
2025 • Climate UX Interest Group
Kate Towsey
Participant Recruitment and Management Tools
2026 • Advancing Research 2026
Gold
Kim Lenox
Leading Distributed Global Teams
2019 • Enterprise Community
Peter Boersma
How to Define and Maintain a DesignOps Roadmap
2023 • DesignOps Summit 2023
Gold
Marc Rettig
The Urge to Migrate
2026 • Rosenfeld Community
Nicole Aleong
What UX research can learn from other research practices [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Ana Ferreira
Designing Distributed: Leading Doist’s Fully Remote Design Team in Six Countries
2024 • DesignOps Summit 2020
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Jorge Arango
AI as Thought Partner: How to Use LLMs to Transform Your Notes (3rd of 3 seminars)
2024 • Rosenfeld Community

More Videos

Nicole Bergstrom

"If you want to work with Hilton, you have to bring accessibility to the table like security and privacy."

Nicole Bergstrom Anna Cook Kate Kalcevich Saara Kamppari-Miller

AccessibilityOps: Moving beyond “nice to have”

September 19, 2024

Johanna Kollmann

"Design strategy focuses on providing a framework for developing design solutions that support a given product strategy, whereas product strategy addresses the business model, the roadmap, and prioritizing requirements - Frank Guo."

Johanna Kollmann

Insights-Driven Product Strategy: Get your Research to Count

December 6, 2022

Josh Clark

"Assistance interfaces combine open-ended dialogue with the ability to do tasks on your behalf."

Josh Clark Veronika Kindred

Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)

January 29, 2025

Nicole Umphress

"We had a matter of weeks to pivot from an in-person immersive program to an entirely virtual environment accommodating designers from India to California."

Nicole Umphress

Delivering Design Education During a Global Pandemic: Lessons Learned

June 9, 2022

Edgar Anzaldua Moreno

"We used a 0.8 correlation threshold to tightly cluster similar survey responses, balancing cluster count and inclusion."

Edgar Anzaldua Moreno

Using Research to Determine Unique Value Proposition

March 11, 2021

"Good UX is good business."

Product and Design at Bloomberg: A 15-year Evolution

December 6, 2022

JJ Kercher

"Our team mission is to accelerate customer value through exceptional end-to-end experiences."

JJ Kercher

A Roadmap for Maturing Design in the Enterprise

June 15, 2018

Peter Van Dijck

"Domain experts are crucial for tagging data because users might say ‘that’s great,’ but experts can tell it’s totally wrong."

Peter Van Dijck

Building impactful AI products for design and product leaders, Part 2: Evals are your moat

July 23, 2025

Jorge Arango

"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."

Jorge Arango

[Demo] How to re-categorize content at scale using LLMs

June 5, 2024