Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Milan Guenther
The $212 billion ‘so what?’: unlocking impact in development cooperation
2025 • Advancing Service Design 2025
Gold
Nicole Bergstrom
AccessibilityOps: Moving beyond “nice to have”
2024 • DesignOps Community
Paul Ford
New work, new words: A glossary for AI
2026 • Designing with AI 2026
Conference
Christian Bason
Innovating With People: Unleashing the Potential of Civic Design
2021 • Civic Design 2021
Gold
Louis Rosenfeld
Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?
2024 • Rosenfeld Community
Peter Van Dijck
Building new AI skills: Creating outsized UX value with evals
2026 • Designing with AI 2026
Conference
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Mackenzie Cockram
Integrating Qualitative and Quantitative Research from Discovery to Live
2022 • QuantQual Interest Group
Sabrina Mach
How to Design Your Design Operating Model
2021 • DesignOps Summit 2021
Gold
Mike Davidson
Fireside Chat
2022 • Advancing Research 2022
Gold
Brad Peters
Short Take #1: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Sam Proulx
Understanding Screen Readers on Mobile: How And Why to Learn from Native Users
2023 • DesignOps Summit 2023
Gold
Dr. Jamika D. Burge
Theme 3 Intro
2022 • Advancing Research 2022
Gold
Bria Alexander
Day 1 Panel: Up to the Minute: The latest in AI’s impact on UX
2025 • Designing with AI 2025
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold
Bria Alexander
Opening Remarks
2021 • Design at Scale 2021
Gold

More Videos

Aurobinda Pradhan

"Design Ops being pushed under DevOps using Jira or Azure is incompatible with the design playground."

Aurobinda Pradhan Shashank Deshpande

Introduction to Collaborative DesignOps using Cubyts

September 8, 2022

Bria Alexander

"There is no need to take notes; session notes, sketchnotes, resource lists, videos, and decks will be shared."

Bria Alexander

Opening Remarks

October 1, 2021

Anil Dash

"Many predictable harms in social media, gig economy, and Web3 were foreseen but not prevented because design lacked the needed power."

Anil Dash

Designing with power in the age of AI

June 10, 2026

Peter Boersma

"It helps if contributing to design ops initiatives is part of people's performance reviews or career ladder criteria."

Peter Boersma

How to Define and Maintain a DesignOps Roadmap

October 3, 2023

Jane Davis

"The hardest part of any researcher's job is translating insights into action."

Jane Davis

Strategic Shifts and Innovations in User Research: Navigating Challenges and Opportunities

March 11, 2025

Dave Hora

"Ten years ago, I did not know that we could do research as a job."

Dave Hora

A Research Skills Evolution

March 10, 2021

Mark Interrante

"The challenge of prioritization increases when decisions are adaptive, systemic, and dynamic."

Mark Interrante Harry Max

AI for Prioritization (3rd of 3 seminars)

July 11, 2024

Kavana Ramesh

"Captions and subtitles were initially designed for people who are deaf or hard of hearing but over 50% of Americans watch content with subtitles."

Kavana Ramesh

Meaningful inclusion: Practicing accessibility research with confidence

September 24, 2024

Sheryl Cababa

"You don’t need a giant system map; even two or three causal loops can reveal root causes and align stakeholders."

Sheryl Cababa

Expanding your Design Lens with Systems Thinking

March 28, 2023