Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews
Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Gabriela Barneva
Operationalizing Inclusive Design in Service Design
2025 • Advancing Service Design 2025
Conference
Randolph Duke II
War Stories LIVE! Randy Duke II
2020 • Advancing Research 2020
Gold
Maish Nichani
Sparking a Service Excellence Mindset at a Government Agency
2021 • Civic Design 2021
Gold
Deanna Zandt
The Unspoken Complexity of “Self-Care” with Deanna Zandt
2022 • Civic Design Community
Zariah Cameron
Streamlining an Inclusive Design Practice
2023 • DesignOps Summit 2023
Gold
Daniela Magaña Flores
Ahead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
2022 • Advancing Research 2022
Gold
Peter Van Dijck
Hands-on AI #2: Understanding evals: LLM as a Judge
2025 • Rosenfeld Community
Louis Rosenfeld
Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?
2024 • Rosenfeld Community
Alfred Kahn
A Seat at the Table: Making Your Team a Strategic Partner
2023 • Design in Product 2023
Gold
Mike Oren
Why Pharmaceutical's Research Model Should Replace Design Thinking
2023 • Advancing Research 2023
Gold
Victor Udoewa
Radical Participatory Research: Decolonizing Participatory Processes
2022 • Advancing Research 2022
Gold
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps Summit 2024
Gold
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 3: Understand AI architectures: RAG, Agents, Oh My!
2025 • Rosenfeld Community
Brennan Hartich
Communicating and Establishing DesignOps as a New Function
2018 • DesignOps Summit 2018
Gold
Sarah Fathallah
Beyond insights: Rethinking the role of researchers as stewards of organizational wisdom
2025 • Advancing Research 2025
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • DesignOps Summit 2023
Gold

More Videos

Sam Proulx

"Disability is the only identity that all of us will probably adopt at some point during our lives."

Sam Proulx

Accessibility: An Opportunity to Innovate

June 8, 2022

Catherine Blizzard

"Some managers felt undermined because they were part of the delivery team rather than just overseeing it."

Catherine Blizzard

Using Integrated Insight to Drive Growth

March 10, 2022

Catherine Dubut

"No one really knows what to expect anyway, so failure goes largely unnoticed. It's better to be transparent about the mess."

Catherine Dubut

Bridging Physical and Digital Spaces: Approaches to Retail Service Design

March 18, 2021

Ellie Krysl

"The design planning and management tool pulls together all necessary design reference material and deliverables, focusing on actively tracked info and their relationships."

Ellie Krysl Jon Fukuda

Planned Right. Managed Right. Designed Right.

June 6, 2023

Marisa Bernstein

"If it doesn’t work for you, it doesn’t work."

Marisa Bernstein

It Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing

December 9, 2021

Joshua Graves

"Psychological safety is key—if folks aren't in the right state, the conversation isn't productive."

Joshua Graves

We Need To Talk: Managing Ludicrous Requests at Work (Part 3 of 3)

May 12, 2025

Emily Williams

"Sometimes you just have to be a little bit bold and be willing to use your power and privilege to advocate for what you know to be true."

Emily Williams Nora Fiore

When UX Research and Institutional Racism Collide: A Case Study

March 12, 2021

Magdalena Zadara

"We treated user research and journey mapping as non-negotiable basics, even when pressured for rapid delivery."

Magdalena Zadara

Zero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late

November 16, 2022

Patrizia Bertini

"We have immature teams where design is still seen as aesthetic rather than strategic experience functionality."

Patrizia Bertini Alexandra Mengoni León

Pushing DesignOps’ Influence into New Global Markets

September 9, 2022