Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • •

    Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • •

    Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • •

    Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • •

    In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • •

    Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • •

    Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • •

    Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • •

    Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • •

    Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • •

    Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Cennydd Bowles
Day 1 Panel
2024 • Designing with AI 2024
Gold
Dana Bishop
2022: The Year UX Demonstrates its Business Impact
2022 • Advancing Research 2022
Gold
Dane DeSutter
Keeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
2024 • Advancing Research 2024
Gold
Beth Chappell
Leading through ambiguity: Supporting a design team relearning their craft
2026 • Designing with AI 2026
Gold
Sam Proulx
Everything You Ever Wanted to Know About Screen Readers
2021 • Design at Scale 2021
Gold
Marina Martin
Lives on the Line: The Stakes of UX at the Scale of Government
2018 • Enterprise Experience 2018
Gold
Chris Geison
What is Research Strategy?
2021 • Advancing Research 2021
Gold
Nick Lewis
Designing and building low-carbon websites independently
2025 • Climate UX Interest Group
Eric Shumake
Diagnosis UX: Building Influence in Healthcare Design
2026 • Rosenfeld Community
Tim Frick
The journey of building a sustainable design practice
2025 • Climate UX Interest Group
Rebecca Gimenez
Work in Progress: Service Design at Airbnb
2024 • Advancing Service Design 2024
Gold
Daniel Korczynski
Why AI Is Bad at Research (and how to make it actually useful)
2026 • Advancing Research 2026
Gold
Caroline Jarrett
Have fun with statistics?
2024 • Rosenfeld Community
Erin Weigel
UX Lessons from running more than 1,200 A/B Tests
2024 • Rosenfeld Community
Christian Crumlish
AMA with Christian Crumlish, author of Product Management for UX People
2022 • Enterprise Community
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Advancing Research 2023
Gold

More Videos

"It’s important not to fall in love with a feature but stay open to new information and feedback."

Product and Design at Bloomberg: A 15-year Evolution

December 6, 2022

Anna Avrekh

"You don’t know how hard it is until you’re leading a user research session yourself."

Anna Avrekh

User Research, Design, and Product - A Love Story

March 11, 2021

Kristen Honey

"When you’re smashing silos, the first wall you try to break through always gets bloody."

Kristen Honey

"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design

November 18, 2021

Renee Bouwens

"Every report should say this research is tied to this KPI, here’s the outcome we want to address."

Renee Bouwens

Landing Product Impact: Aligning Research as a Foundational Driver for Delivering the World’s Best Products

December 15, 2023

Husani Oakley

"Talent is everywhere. You find talent when you look for it without skepticism and bias."

Husani Oakley

Bias Towards Action: Building Teams that Build Work

June 14, 2018

David Conrad

"Data quality is multifaceted, subjective, and can't be reduced to simple profiles alone."

David Conrad

The Feeling of Data

September 14, 2023

Lukas Moro

"We need to factor in the specifics of the social situation when designing interaction models and gestures for smart glasses."

Lukas Moro

“Feels Like Paper!”: Interfacing AI through Paper

June 11, 2025

Adam Cutler

"We don’t call them UX designers or front-end designers or visual designers. They’re all just designers."

Adam Cutler Karen Pascoe Ian Swinson Susan Worthman

Discussion

June 8, 2016

Louis Rosenfeld

"It’s better to err on the side of kindness and emotional support toward colleagues who might be struggling."

Louis Rosenfeld

Welcome / Housekeeping

June 7, 2023