Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Kevin Bethune
Gatekeepers and Servant Leadership
2020 • DesignOps Community
Ariel Kennan
Building a Design Culture
2017 • Enterprise Experience 2017
Gold
Sofia Quintero
Beyond Tools: The Messy Business of Implementing Research Repositories
2022 • Advancing Research 2022
Gold
Christian Crumlish
Afternoon Insights Panel
2022 • Design in Product 2022
Gold
Heidi Trost
When AI Becomes the User’s Point Person—and Point of Failure
2025 • Rosenfeld Community
Christian Crumlish
Morning Insights Panel
2022 • Design in Product 2022
Gold
Sha Hwang
The Lost Year
2021 • Design at Scale 2021
Gold
Nicole Aleong
What UX research can learn from other research practices [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
The AI + Design wave has arrived: Making sense of the 2026 AI + Design Report
2026 • Rosenfeld Community
Alla Weinberg
Cross-Functional Relationship Design
2022 • Design in Product 2022
Gold
Jim Kalbach
Jazz Improvisation as a Model for Team Collaboration
2019 • Enterprise Experience 2019
Gold
Brenna Fallon
Learning Over Outcomes
2019 • DesignOps Summit 2019
Gold
Jon Temple
Panel: Stacks, Security, and Stakeholders: The Hidden Work of UXR Tool Procurement
2026 • Advancing Research 2026
Gold
Iulia Cornigeanu
QuantQual Book Club: Small Data
2024 • QuantQual Interest Group
Maish Nichani
Sparking a Service Excellence Mindset at a Government Agency
2021 • Civic Design 2021
Gold
Jen Cardello
Standardizing Product Merits for Leaders, Designers, and Everyone
2018 • Enterprise Experience 2018
Gold

More Videos

Megan Blocker

"Demonstrate business value of less extractive, participatory research methods to make ethical practices easier to adopt."

Megan Blocker Marieke McCloskey Renee Reid

Positioning insight: Structuring teams, roles and careers for a changing research landscape

March 13, 2025

Sam Proulx

"Mobile screens have forced designers to create simplified interfaces without reducing features."

Sam Proulx

Mobile Accessibility and You

June 9, 2022

Carol Scott

"If you don't have good usability to begin with, being trauma-informed is not going to work out."

Carol Scott Melissa Eggleston

Avoid Harming Your Team and Users: Promoting Care and Brand Reputation with Trauma-Informed UX Practices

February 5, 2025

Sam Yen

"Introducing design gates allowed us to finally stop the shipment of bad products."

Sam Yen

Driving Organizational Change Through Design? Do more of this and less of that

June 9, 2017

Rachel Posman

"Designers deserve support and infrastructure so they can really excel in their work."

Rachel Posman John Calhoun

"Ask Me Anything" with Rachel Posman and John Calhoun, Authors of the Upcoming Rosenfeld Book, The Design Conductors

September 25, 2024

Anna Avrekh

"Diversity is not just about how people look or sound, but also about their capabilities and their place on introversion-extraversion scales."

Anna Avrekh Amy Jiménez Márquez Morgan C. Ramsey Catarina Tsang

Diversity In and For Design: Building Conscious Diversity in Design and Research

June 9, 2021

Josh Clark

"These intelligent interfaces are collaborative, proactive partners on the user’s journey."

Josh Clark Veronika Kindred

Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)

January 29, 2025

Sam Ladner

"Some of the most mundane objects can signal change."

Sam Ladner

How Research Can Drive Strategic Foresight

March 9, 2022

Lija Hogan

"Being fluent in multiple data languages and understanding behavioral and attitudinal data will be essential."

Lija Hogan Milan Mijatovic Sam Proulx Louis Rosenfeld

Three Years Out: Perspectives on the Near-Term Future of User Research

March 15, 2024