Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Eniola Oluwole
Lessons From the DesignOps Journey of the World's Largest Travel Site
2019 • DesignOps Summit 2019
Gold
Sharon Bautista
Time to Make the Donuts: How User Research Helped Bridge Disparate Teams
2024 • Enterprise Experience 2020
Gold
Ed Mullen
Designing the Unseen: Enabling Institutions to Build Public Trust
2022 • Civic Design 2022
Gold
Maria Giudice
Becoming a Changemaker by Leading with Design
2023 • Advancing Research 2023
Gold
Holly Cole
Panel Discussion: Growing People and Teams
2018 • DesignOps Summit 2018
Gold
Rebecca Topps
Planning and conducting remote usability studies for accessibility
2020 • Advancing Research Community
Kit Unger
Theme 1 Intro
2022 • Design at Scale 2022
Gold
Johnny Michaelsen
Measure Behaviors, Not Results
2026 • Rosenfeld Community
Marc Fonteijn
Increase your confidence, influence, and impact (through a Professional Community)
2024 • Advancing Service Design 2024
Gold
Bria Alexander
Opening Remarks
2024 • Advancing Research 2021
Gold
Sofia Quintero
Beyond Tools: The Messy Business of Implementing Research Repositories
2022 • Advancing Research 2022
Gold
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Natalia Radywyl
Co-Designing New Power in Australia's Public Sector
2022 • Civic Design 2022
Gold
Erin Hauber
Design is Not the Frosting on the Scaled Agile Layer Cake
2019 • DesignOps Summit 2019
Gold
Billy Carlson
Tips to Utilize Wireframes to Tell an Effective Product Story
2023 • Enterprise UX 2023
Gold
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold

More Videos

Laura Schaefer

"Putting accessibility and inclusion in writing makes it real and formal, not just a spoken idea."

Laura Schaefer

DesignOps: A Conduit for Inclusion

September 9, 2022

Saara Kamppari-Miller

"I am one of the lucky ones. I have not lost my job this year and I get to work on inclusion and accessibility every day."

Saara Kamppari-Miller

Theme Three Intro

October 4, 2023

Bethany Brown

"Co-creation only works when it has boundaries."

Bethany Brown

Rewiring operations with service design and AI

November 20, 2025

Andrea Gallagher

"We have eight million prototyping tools but not many tools that help us spend more time in the problem space."

Andrea Gallagher

The Problem Space

May 16, 2019

Jemma Ahmed

"We can no longer rely on interviews, surveys, and usability tests to answer the strategic, complex questions organizations face today."

Jemma Ahmed Megan Blocker Eduardo Ortiz

Redefining the research toolkit: Expanding methodologies for a changing world

March 12, 2025

Dean Broadley

"Don't be a pot plant, be a crab."

Dean Broadley

Not Black Enough to be White

January 8, 2024

Yoel Sumitro

"My research team focused too much on scientific rigor, while design teams improved intuition with reflection and demonstration."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Bria Alexander

"The good vibes you experience are a reflection of the positive collaborative spirit we worked hard to create for you."

Bria Alexander

Theme Two Intro

October 3, 2023

Anna Avrekh

"You don’t know how hard it is until you’re leading a user research session yourself."

Anna Avrekh

User Research, Design, and Product - A Love Story

March 11, 2021