Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews
Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Sheri Byrne-Haber
The Importance of Accessible Design Systems
2024 • DesignOps Summit 2020
Gold
Verónica Urzúa
The B-side of the Research Impact
2021 • Advancing Research 2021
Gold
John Calhoun
Two Sides of the DesignOps Coin: Teams Ops and Product Ops
2024 • DesignOps Summit 2020
Gold
Sara Asche Anderson
Not Your Ordinary Re-Brand: Design's Path to Driving Customer Obsession at Best Buy
2024 • Enterprise Experience 2020
Gold
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Sarah Coyle
Design and Analytics with Sarah Coyle
2020 • DesignOps Community
Michael Polivka
Scaling Design through Relationship Maps
2017 • DesignOps Summit 2017
Gold
Sol Mesz
Hands or Brains? How to Hire for Strategy, Strategically
2024 • Enterprise Experience 2020
Gold
Prayag Narula
HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer
2023 • Advancing Research 2023
Gold
James Lang
If you can design an app, you can design a community
2025 • Rosenfeld Community
Emily Eagle
Can't Rewind: Radio and Retail
2019 • Enterprise Experience 2019
Gold
Greg Petroff
The Compass Mission
2021 • Advancing Research 2021
Gold
Joel Branch
Humanizing AI: Filling the Gaps with Multi-faceted Research
2021 • Advancing Research 2021
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Roberta Dombrowski
Making Research a Team Sport
2022 • Advancing Research 2022
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold

More Videos

Tracy McGoldrick

"It’s like sending around the Rolls Royce with the champagne bottle – a white glove, concierge service for recruiting partners."

Tracy McGoldrick

IBM User Experience Program—The What, Why and How

October 15, 2021

Laine Riley Prokay

"I don’t know what I don’t know – new practitioners may not even realize what questions to ask at first."

Laine Riley Prokay Lisa Gordon

Carving a Path for Early Career DesignOps Practitioners

September 9, 2022

Jorge Arango

"Culture eats strategy for breakfast—integrating cultures is the hardest and most critical part of mergers."

Jorge Arango

Meeting of the Waters: Designing for Successful Inorganic Growth

August 12, 2021

Marisa Bernstein

"If it doesn’t work for you, it doesn’t work."

Marisa Bernstein

It Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing

December 9, 2021

Kate Kalcevich

"If the code doesn’t match the label on screen, voice controls might not recognize the command."

Kate Kalcevich

Designing inclusively with AI

June 5, 2024

Melissa Tsang

"More design feedback now happens in real time on Slack instead of waiting a whole week."

Melissa Tsang

From Insights to Action: Driving Business Values through DesignOps

January 8, 2024

Victor Udoewa

"In times of great upheaval, possibilities for transformation become more alive."

Victor Udoewa

Theme One Intro

March 27, 2023

Séamus Byrne

"Routine shapes how we collaborate—different teams have their own habits and we need to learn to dance together differently."

Séamus Byrne

Aligning Teams with Choreography

January 8, 2024

Andy Barraclough

"Anyone can do a basic AI summary with zero control, but we offer robust, evidence-backed, scalable chat analysis."

Andy Barraclough Betsy Nelson

From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme

September 23, 2024