Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Kit Unger
Theme 2: Discussion
2024 • Enterprise Experience 2020
Gold
Ben Reason
Making the system visible: The fastest path to better decisions
2025 • Advancing Service Design 2025
Gold
Jack Moffett
SAFe or Sorry?
2019 • Enterprise Community
Indi Young
Thinking styles: Mend hidden cracks in your market
2025 • Rosenfeld Community
Jacqui Frey
Panel Discussion: Integrating DesignOps
2018 • DesignOps Summit 2018
Gold
Anupama Dhareshwar
From blueprint to bot: Designing resilient AI-powered services
2025 • Advancing Service Design 2025
Gold
Laura Schaefer
DesignOps: A Conduit for Inclusion
2022 • DesignOps Summit 2022
Gold
George Aye
Designing a New Social Contract
2026 • Rosenfeld Community
Aletheia Delivre
New Shapes and Emerging Identities for Design Ops
2025 • DesignOps Summit 2025
Gold
Jack Behar
How to Build Prototypes that Behave like an End-Product
2022 • Design in Product 2022
Gold
Rachael Dietkus, LCSW
Leading through the long tail of trauma
2022 • Enterprise Community
Gonzalo Goyanes
Design ROI: Cover a Little, Get a Lot
2022 • DesignOps Summit 2022
Gold
Catt Small
What's Next for ICs: Exploring Staff and Principal Designer Roles
2024 • Rosenfeld Community
George Abraham
Design Systems To-Go: Reimagining Developer Handoff, and Introducing App Builder (Part 2)
2021 • DesignOps Summit 2021
Gold
Laura Weiss
There is No Playbook: Leader as Coach During Challenging Times
2024 • Rosenfeld Community

More Videos

Corey Long

"Always communicate next steps, timelines, and delays honestly with candidates; they will appreciate it and stay engaged."

Corey Long

Hiring in DesignOps: A Critical Study on How to Hire and Get Hired

September 23, 2024

David Eisner

"It’s giving this kind of shared place where we’re all working, as opposed to previously designers are in Figma, devs in VS Code."

David Eisner Changying (Z) Zheng Amelia Wattenberger Elyse Holladay

From prototype to production: Vibe coding design for real engineering systems

June 10, 2026

Anupama Dhareshwar

"It’s important to establish feedback loops so each organizational layer learns from the others continuously."

Anupama Dhareshwar

From blueprint to bot: Designing resilient AI-powered services

November 19, 2025

Mansi Gupta

"Safety is a fundamental human need which is often taken for granted and maybe that’s why it’s forgotten about."

Mansi Gupta

Drawing from Feminist Practice to Make Inclusive Design Operational

September 9, 2022

Liam Thurston

"Our skills matrix became the center of gravity for our practice—it started writing job descriptions and guiding recruitment."

Liam Thurston

Why Your Design Team Is Quitting, And How To Fix It

June 10, 2022

Emily Danielson

"Be extra patient and gentle with people, especially if they don’t pick up your call after a disaster."

Emily Danielson

“I mean, I can lift a shovel”: Design Skills in Disaster Response

June 9, 2022

Juhan Sonin

"You have to find your needle focus and go after it."

Juhan Sonin

Design Now! The Agenda for Action

September 4, 2025

Lada Gorlenko

"We never compromise on the business goals, but how we achieve them can be flexible and pragmatic."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Holly Cole

"The true problem was that two companies weren’t communicating, it had nothing to do with the UX team itself."

Holly Cole

Understanding Experiences: When you have to do more than work

November 8, 2018