Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews
Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Robert Reimann
Taming Design Complexity with UX Models
2017 • Enterprise Experience 2017
Gold
Tamara Hale
War Stories LIVE! Tamara Hale
2020 • Advancing Research 2020
Gold
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Theresa Neil
Just Build Me a Dashboard!
2019 • Enterprise Community
John Cutler
Prioritization for designers and product managers (1st of 3 seminars)
2024 • Rosenfeld Community
Shanti Mathew
Civic Design at Scale: Introducing the Public Policy Layer Cake
2021 • Civic Design 2021
Gold
Dean Broadley
Not Black Enough to be White
2024 • DesignOps Summit 2020
Gold
Deanna Washington
Connecting the Ops: Plenary Panel and Closing Circle
2022 • DesignOps Summit 2022
Gold
Jemma Ahmed
Theme Three Intro
2023 • Advancing Research 2023
Gold
Charlotte Lee
Theme 1 Intro
2021 • Civic Design 2021
Gold
John Maeda
Making Sense of Enterprise UX
2016 • Enterprise UX 2016
Gold
Rusha Sopariwala
Remote, Together: Craft and Collaboration Across Disciplines, Borders, Time Zones, and a Design Org of 170+
2022 • Design at Scale 2022
Gold
Brad Peters
Short Take #1: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Tina Weisser
When AI Agents Meet Reality. Service Design Lessons from a Pilot
2026 • Rosenfeld Community
Joi Freeman
A New Vantage Point: Building a Pipeline for Multifaceted Research(ers)
2020 • Advancing Research 2020
Gold
Rich Mironov
How Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
2022 • Design in Product 2022
Gold

More Videos

Jules Monza

"It’s the same thing with spreadsheets or work back plans; if you’re spending all your time on process and not the work, that’s too much overhead."

Jules Monza

Use These Words and Count These Things

September 25, 2024

Dharani Perera

"We moved a months-long deadlock in a matter of weeks by putting service design practices in place."

Dharani Perera

The mandala of service design: unlocking alignment and action through service design

November 20, 2025

Dominique Ward

"We’re really in the game of behavioral change—planting seeds today so teams move naturally towards the future."

Dominique Ward

The Most Exciting Time for DesignOps is Now

September 8, 2022

Uday Gajendar

"Each conference theme acts like a mini conference exploring a deep aspect of designing at scale."

Uday Gajendar

Theme 1: Introduction

June 9, 2021

Jane Reid

"It’s okay to be honest with participants that you’re not the right person to help them and to signpost support respectfully."

Jane Reid Janice Hannaway

Self-care in User Research

April 2, 2020

Aditi Ruiz

"This work does not fit our product program."

Aditi Ruiz Christian Crumlish Farid Sabitov

A PM State of Mind: Empathy Mapping Your Product Manager, Pt. 1

December 6, 2022

Stephen Pollard

"This client project is very deliberate and explicit in focusing on culture and values."

Stephen Pollard

Closing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?

November 7, 2017

Dr. Jamika D. Burge

"Organizational change management requires as much care and effort as executing good research."

Dr. Jamika D. Burge Robert Fabricant Peter Merholz Noam Segal Teresa Torres

A Genuine Conversation about the Future of UX Research

March 20, 2024

Malini Rao

"Legacy applications linger because changing them is risk-ridden and a major undertaking."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021