Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews
Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Neil Barrie
Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
2024 • Advancing Research 2024
Gold
Julie Norvaisas
Back to basics, or start from scratch?
2025 • Advancing Research 2025
Gold
Kit Unger
Theme 2: Introduction
2021 • Design at Scale 2021
Gold
Mike Davidson
Fireside Chat
2022 • Advancing Research 2022
Gold
Uday Gajendar
Theme Four Intro
2023 • Enterprise UX 2023
Gold
Sam Ladner
Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX
2016 • Enterprise UX 2016
Gold
Nicole Umphress
Delivering Design Education During a Global Pandemic: Lessons Learned
2022 • Design at Scale 2022
Gold
Clemens Janssen
Efficiently Scaling Research as a Team of One
2023 • Advancing Research 2023
Gold
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Justin Entzminger
Risk and Reward: How to Diversify the Field of Civic Innovators and Designers
2022 • Civic Design 2022
Gold
Ilana Lipsett
Anticipating Risk, Regulating Tech: A Playbook for Ethical Technology Governance
2021 • Civic Design 2021
Gold
Benjamin Real
Maturity Models: A Core Tool for Creating a DesignOps Strategy
2021 • DesignOps Summit 2021
Gold
Alexia Cohen
Increasing Health Equity and Improving the Service Experience for Under-Served Latine Communities in Arizona
2024 • Advancing Service Design 2024
Gold
Maverick Chan
From Doodle to Demo: AI as Our Storytelling Partner
2025 • Rosenfeld Community
John Calhoun
Have we Reached Our Peak? Spotting the Next Mountain For DesignOps to Climb
2021 • DesignOps Summit 2021
Gold
Dorelle Rabinowitz
The Magic Word is Trust
2018 • Enterprise Experience 2018
Gold

More Videos

Sam Ladner

"You have to be careful not to be too weird; mental models differ internationally."

Sam Ladner

How Research Can Drive Strategic Foresight

March 9, 2022

Bria Alexander

"Even though we’re virtual, we are still a community and obligated to treat each other with kindness and respect."

Bria Alexander Louis Rosenfeld

Welcome

September 8, 2022

Christian Bason

"Profit making and shareholder value have been a huge part of the problem if we want a sustainable world."

Christian Bason

Expand—Rethinking Design for Public Challenges

September 14, 2022

Sam Proulx

"Mobile screens can be angled and positioned easily, which helps if you struggle with glare or neck positioning."

Sam Proulx

Mobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World

November 17, 2022

Maria Giudice

"An ending occurs when the reality of your present moment shifts or changes all together."

Maria Giudice

Empowering change: Reigniting purpose, passion and impact in research

March 13, 2025

Kate Towsey

"People don't learn when they're rushed, stressed, or don't trust the source of the knowledge."

Kate Towsey

Ask Me Anything (AMA) with Kate Towsey

April 2, 2025

Sam Proulx

"If you want an accessible app on iOS and Android, using the standard native APIs and controls will get you there."

Sam Proulx

Mobile Accessibility and You

June 9, 2022

Bob Baxley

"I don’t think it’s taking the joy out, it’s helping people find where the joy is."

Bob Baxley

Leading with Design Operations Past and Present

December 19, 2019

Jemma Ahmed

"Our job is to make sure the people making decisions actually listen."

Jemma Ahmed

Theme 1 Intro

March 10, 2026