Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews
Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Matt Duignan
HITS, Microsoft's internal human insight system: From research library to living body of knowledge
2019 • Advancing Research Community
Nova Wehman-Brown
We've Never Done This Before
2019 • Enterprise Experience 2019
Gold
Jen Cardello
Learning Velocity—The Insights Speedometer
2021 • Advancing Research Community
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • DesignOps Summit 2022
Gold
Jemma Ahmed
Democratization: Working with it, not against it [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Boon Yew Chew
Making Sense of Systems—and Using Systems to Make Sense of the Enterprise
2023 • Enterprise UX 2023
Gold
Charlotte Vorbeck
Pipeline to Civic Design
2021 • Civic Design 2021
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Ilana Lipsett
Anticipating Risk, Regulating Tech: A Playbook for Ethical Technology Governance
2021 • Civic Design 2021
Gold
Jerome “Axle” Brown
How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon
2021 • Advancing Research 2021
Gold
Rachael Greene
Building a Design Ops Practice that Really Works (Most of the Time)
2025 • DesignOps Community
Jamika Burge
Embracing change: Navigating shifting landscapes with compassion and agency
2025 • Advancing Research 2025
Gold
Lisa Gironda
Opener: Chief of Staff–An unexpected journey
2024 • DesignOps Summit 2020
Gold
Angelos Arnis
Navigating the Rapid Shifts in Tech's Turbulent Terrain
2023 • DesignOps Summit 2023
Gold
Robin Beers
Navigating organizational systems: Rethinking researcher’s role in driving change
2025 • Advancing Research 2025
Gold
Jon Fukuda
The Big Question about Innovation: A Panel Discussion
2024 • DesignOps Summit 2024
Gold

More Videos

Kristin Skinner

"The plan always changes, but we need to be ready."

Kristin Skinner

Opening Keynote: Org Design for Design Orgs

November 6, 2017

Jonathan Fairman

"The product is no longer fixed; after the first interaction, it diverges user by user."

Jonathan Fairman Kevin Johnson

Integrating generative AI into enterprise products: A case study from dscout

June 5, 2024

Peter Van Dijck

"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."

Peter Van Dijck

Building the Rosenbot

June 4, 2024

Nancy Douyon

"The nobility complex is when we in the West unintentionally create solutions without accounting for our own biases."

Nancy Douyon

We'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex

June 15, 2018

Andreas Huebner

"Our user base is really fun and complex, with different preferences that require nuanced automation approaches."

Andreas Huebner Amy Takata Craig Brookes

What Is It Like To Be Part of The UX Team at Compass?

March 11, 2021

Barb Spanton

"Regulations aren’t obstacles to dismiss but tools to respect and embrace that help us protect grandma."

Barb Spanton

Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'

June 10, 2022

Erin May

"Stakeholders aren’t ready to write their own briefs or manage recruitment, so it ended up being a hybrid model initially."

Erin May Roberta Dombrowski Laura Oxenfeld Brooke Hinton

Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org

March 10, 2022

Shan Shen

"Start small, try to learn as you go, and set realistic expectations to avoid being humbled when things go live."

Shan Shen

Translating UX Terms into Business Contexts

November 29, 2023

Charlotte Vorbeck

"Build a culture and environment where diverse people feel comfortable and want to come to work every day."

Charlotte Vorbeck Shahzma Esmail Edward Alton Sarah McArthur Ariel Kennan

Pipeline to Civic Design

December 9, 2021