Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Mike Davidson
Fireside Chat
2022 • Advancing Research 2022
Gold
Stephen Pollard
Closing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?
2017 • DesignOps Summit 2017
Gold
Jason Mesut
Shaping design, designers and teams
2018 • DesignOps Summit 2018
Gold
Erin Weigel
Get Your Whole Team Testing to Design for Impact
2024 • Rosenfeld Community
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 1: The new product journey
2025 • Rosenfeld Community
Liam Thurston
Why Your Design Team Is Quitting, And How To Fix It
2022 • Design at Scale 2022
Gold
Katie Hansen
Finding the unknown in the known: Harnessing meta-analysis and literature review
2025 • Advancing Research 2025
Gold
Bruce Gillespie
Learning from journalism: Balancing impactful communication with compassionate storytelling
2025 • Advancing Research 2025
Gold
Katie Hansen
Experimental research: techniques for deep, psychology-driven insights
2025 • Advancing Research 2025
Gold
Dan Willis
Filling the Void
2018 • DesignOps Summit 2018
Gold
Emilia Åström
Unlock Your Team’s Intelligence with Collaboration Design
2022 • Design at Scale 2022
Gold
Russell Blair
Killing the blank page
2024 • Designing with AI 2024
Gold
Ariel Kennan
Building a Design Culture
2017 • Enterprise Experience 2017
Gold
Sam Proulx
Understanding Screen Readers on Mobile: How And Why to Learn from Native Users
2023 • DesignOps Summit 2023
Gold
Sarit Geertjes
People, not Petri Dishes: Stories from a Research Recruiter
2019 • DesignOps Community
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold

More Videos

Jerome “Axle” Brown

"Combining synchronous and asynchronous communication within one session gets us from awareness to action."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Michal Anne Rogondino

"Our modern way of life is actually really dependent on satellites."

Michal Anne Rogondino

Saving Outer Space: The First UX Design System for Our Nation’s Satellites

January 8, 2024

Sarah Fathallah

"Participation is an ethical commitment but should be complemented by an ethic of care, valuing people’s time and being trauma responsive."

Sarah Fathallah

A Typology of Participation in Participatory Research

March 28, 2023

John Cutler

"A story lets people assign the meaning that's powerful to them and then think about the decisions they need to make and align."

John Cutler

The Alignment Trap

November 29, 2023

Mansi Gupta

"Women are living nonlinear lives—career breaks, health cycles—that standard designs do not account for."

Mansi Gupta

Women-Centric Research: What, Why, How

March 29, 2023

Karen Pascoe

"If you have somebody on your team for three years, for five years, that’s amazing from a longevity perspective given enterprise legacy and complexity."

Karen Pascoe

Developing Experience Teams and Talent in the Enterprise

June 8, 2016

Smitha Papolu

"The happier their team is, the better the work they deliver for me will be."

Smitha Papolu Nova Wehman-Brown Melissa Schmidt Adam Menter

Theme 3 Discussion

January 8, 2024

Sam Proulx

"If you make an app that users only use once a year, don’t expect them to learn dozens of custom shortcuts."

Sam Proulx

Everything You Ever Wanted to Know About Screen Readers

June 11, 2021

Bria Alexander

"Sponsor sessions are not sales pitches; they are super similar to the quality of the main conference session."

Bria Alexander

Opening Remarks

October 1, 2021