Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Advancing Research 2022
Gold
Chris Hammond
Embedding sustainability into enterprise design and development: A journey towards "sustainability consciousness"
2025 • Climate UX Interest Group
Eniola Oluwole
Lessons From the DesignOps Journey of the World's Largest Travel Site
2019 • DesignOps Summit 2019
Gold
Bria Alexander
Day 3 Welcome
2024 • DesignOps Summit 2024
Gold
Alana Washington
Theme 3 Intro
2021 • DesignOps Summit 2021
Gold
Kyria Stephens
Power to Heal: Civic Design in the Aftermath of Tragedy
2022 • Civic Design 2022
Gold
Bilan Hashi
The Tension Between Story Collecting and Story Telling in Research
2021 • Advancing Research 2021
Gold
Justin Entzminger
Risk and Reward: How to Diversify the Field of Civic Innovators and Designers
2022 • Civic Design 2022
Gold
Jack Moffett
SAFe or Sorry?
2019 • Enterprise Community
Ben Davies
Expert Panel: The Principles of Research Repository Design
2022 • Advancing Research 2022
Gold
Cheryl Platz
Collaborative Creativity through Improv
2018 • DesignOps Summit 2018
Gold
Matt Bernius
Trauma-informed Research: A Panel Discussion
2021 • Advancing Research Community
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Civic Design 2022
Gold
Ana Ferreira
Designing Distributed: Leading Doist’s Fully Remote Design Team in Six Countries
2024 • DesignOps Summit 2020
Gold
Alicia D. Johnson
Disasters and the 21st Century
2021 • Civic Design 2021
Gold
John Calhoun
Meters, Miles, and Madness: New Frameworks to Measure the (Elusive) Value of DesignOps
2024 • DesignOps Summit 2024
Gold

More Videos

Robin Beers

"Learning is the new knowing, and researchers are catalysts of organizational learning."

Robin Beers

Navigating organizational systems: Rethinking researcher’s role in driving change

March 13, 2025

Joanna Vodopivec

"Video parties allow many teams to engage and observe learnings in a more compelling and digestible way than slides."

Joanna Vodopivec Prabhas Pokharel

One Research Team for All - Influence Without Authority

March 9, 2022

Karen Pascoe

"Most employees take at least 90 days to onboard and another 90 days to become productive — that’s half a year before you see impact."

Karen Pascoe

Developing Experience Teams and Talent in the Enterprise

June 8, 2016

Matteo Gratton

"In our product, no manager can see more data than any designer; transparency is our red line."

Matteo Gratton

Can Data and Ethics Live Together?

October 1, 2021

Molly Fargotstein

"UX research marketing is the strategy behind and implementation of intentional effective promotion and communication of who UX research is and what UX research does."

Molly Fargotstein

Multipurpose Communication & UX Research Marketing

September 12, 2019

Steve Chaparro

"Simplicity rarely loses to complexity in the public square—it’s about meeting people where they are."

Steve Chaparro

Bringing Into Alignment Brand, Culture and Space

August 13, 2020

Andrew Michael

"Embedded squads build domain expertise but risk losing knowledge if someone leaves, while centralized teams help reduce duplicated effort."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Brian T. O’Neill

"You need to prototype with real or believable data so the data doesn’t change the context of use."

Brian T. O’Neill Maria Cipollone Luis Colin Manuel Dahm Mike Oren

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

February 18, 2022

Michaela Mora

"When you start concept testing, be clear about what parts of the concept describe the what, the why, and the how."

Michaela Mora

Advanced Concept Testing Approaches To Guide Product Development and Business Decisions

March 11, 2022