Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Bria Alexander
State of DesignOps Panel
2021 • DesignOps Summit 2021
Gold
Melinda Belcher
Bridging the Gap: Making the Most of the Differences Between Agency and Enterprise
2024 • Enterprise Experience 2020
Gold
Chris Geison
Theme 1 Intro
2022 • Advancing Research 2022
Gold
David Cronin
The GE Design System and Thoughts about Craft at Scale
2015 • Enterprise UX 2015
Gold
Joe Elmendorf
Shitty Maps: A Trojan Horse for Sensemaking
2026 • Rosenfeld Community
Stephen Pollard
Closing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?
2017 • DesignOps Summit 2017
Gold
Dagmara Kukawka
Tiny team, moonshot impact: Democratizing research across continents
2026 • Advancing Research 2026
Gold
Alissa Briggs
How to Coach Enterprise Experimentation
2015 • Enterprise UX 2015
Gold
Lori Muszynski
Keeping Design Weird
2023 • DesignOps Summit 2023
Gold
Allan Lowson
Rehashing the Double Diamond: Collaborating across functions with AI-assisted prototyping
2026 • Designing with AI 2026
Conference
Frances Yllana
D.E.A.R.R. Diaries (Discipline, Experience, Architecture, Reflection + Revolution)
2022 • Civic Design 2022
Gold
Juhan Sonin
Design Now! The Agenda for Action
2025 • Rosenfeld Community
Bria Alexander
Theme Two Intro
2022 • DesignOps Summit 2022
Gold
Dave Malouf
The Past, Present, and Future of DesignOps: a 2-part DesignOps Community Call (Part 1)
2022 • DesignOps Community
Helen Armstrong
Augment the Human. Interrogate the System.
2023 • Enterprise UX 2023
Gold
Kwabena Opoku
Methodological toolkit for unique research impact
2026 • Advancing Research 2026
Gold

More Videos

Ana Maria Montero Barrantes

"We may not have tech prowess but our sticky notes, card sorts, and test reports are our real superpowers."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Satyam Kantamneni

"When you compete with your users and not your competition, that's the essence of what we’re trying to do."

Satyam Kantamneni

Do You Have an Experience Vision?

March 23, 2023

Peter Van Dijck

"If you don’t jump in and learn, you’ll spend your time reading stuff on it and not really understand the tech."

Peter Van Dijck Louis Rosenfeld

Coffee with Lou #4: Taking a Peek Under the Rosenbot's Hood

June 14, 2024

Molly Fargotstein

"Measuring the success of marketing tactics is starting to become a focus; right now it’s fly-by-the-seat-of-our-pants but we want to quantify and prove impact."

Molly Fargotstein

Multipurpose Communication & UX Research Marketing

September 12, 2019

Husani Oakley

"X is going to change everything. Sometimes it’s true, and sometimes it’s just hype."

Husani Oakley

Theme Three Intro

June 6, 2023

Steve Baty

"Cross-training helps teams feel more capable and less afraid of losing their jobs during transitions, creating less resistance to change."

Steve Baty Richard Dalton Maria Giudice Harry Max

Discussion

June 9, 2016

Theresa Neil

"EMRs and EHRs have terrible user experience and are ripe for design innovation, similar to FinTech years ago."

Theresa Neil

Designing for Wellness: Specializing in Healthcare

May 22, 2024

Louis Rosenfeld

"Most bad panels feel like a bunch of people showing up at the last minute without shared goals or preparation."

Louis Rosenfeld Jemma Ahmed Christian Crumlish Uday Gajendar Chris Geison

Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?

May 2, 2024

Sofia Quintero

"Introducing change is introducing stress and fear and feelings of rejection."

Sofia Quintero

Beyond Tools: The Messy Business of Implementing Research Repositories

March 10, 2022