Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Catt Small
Craft a Vision that Actually Gets Shipped
2026 • Rosenfeld Community
Chris Hammond
Embedding sustainability into enterprise design and development: A journey towards "sustainability consciousness"
2025 • Climate UX Interest Group
Mitchell Bernstein
Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
2021 • DesignOps Summit 2021
Gold
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • DesignOps Summit 2022
Gold
Mila Kuznetsova
How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices
2022 • Advancing Research 2022
Gold
Brigette Metzler
Scaling ResearchOps: Helping Researchers do Their Best Work
2020 • Advancing Research 2020
Gold
Jim Kalbach
Jobs To Be Done
2021 • Enterprise Community
Dave Malouf
The Future of DesignOps
2024 • DesignOps Summit 2020
Gold
Anna Nguyen
Why Our Voice of the Customer is Better Than Yours
2022 • Advancing Research 2022
Gold
Tina Weisser
When AI Agents Meet Reality. Service Design Lessons from a Pilot
2026 • Rosenfeld Community
Josina Vink
Navigating the pitfalls of systems thinking in service design
2024 • Advancing Service Design 2024
Gold
Lija Hogan
Doing more with more: Lessons from the Front Lines of Democratization
2022 • Advancing Research 2022
Gold
Bilan Hashi
The Tension Between Story Collecting and Story Telling in Research
2021 • Advancing Research 2021
Gold
Natalia Radywyl
Co-Designing New Power in Australia's Public Sector
2022 • Civic Design 2022
Gold
Carol Massa
Designing Health: Integrating Service Design, Technology, and Strategy to Transform Patient and Clinician Experiences
2024 • Advancing Service Design 2024
Gold
Frank Duran
Partnership Playbook: Lessons Learned in Effective Partnership
2024 • Enterprise Experience 2020
Gold

More Videos

Charlotte Vorbeck

"Build a culture and environment where diverse people feel comfortable and want to come to work every day."

Charlotte Vorbeck Shahzma Esmail Edward Alton Sarah McArthur Ariel Kennan

Pipeline to Civic Design

December 9, 2021

Daniel Orbach

"What if all designers were design operators?"

Daniel Orbach

Zero to One: Co-Creating Operating Models with your Team

September 23, 2024

Marc Majers

"Interruptions reveal things like auto-save that you never thought about before."

Marc Majers Tony Turner

Interrupted UX - Add A Dose of Reality To Usability Testing

March 11, 2022

Brendan Jarvis

"Being here alive at this moment is a statistical miracle—we must make the most of it."

Brendan Jarvis

It was the Best of Times. It was the Worst of Times.

September 25, 2024

Christian Crumlish

"This is for me an amazing opportunity to see these ideas explored by brilliant people with a wonderful community."

Christian Crumlish

Introduction by our Conference Chair

December 6, 2022

Mandy Drew

"Infinite scroll is behavioral cocaine sprinkled all over our interfaces, triggering dopamine hits that keep us endlessly scrolling."

Mandy Drew

What Role(s) Can Research Play in Responsible Design?

March 11, 2021

Bob Baxley

"The theme today, 'better together,' focuses on partnering with others to transform."

Bob Baxley

Theme 4: Intro

January 8, 2024

Eric Shumake

"If you get 10 people to follow, that’s a good sign your business has traction."

Eric Shumake

An AMA on UX's Role in Healthcare

April 22, 2026

Jorge Arango

"I see these AI tools as augmenting human work, letting us focus on what matters most, not replacing humans."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024