Rosenverse
What co-speech gestures reveal about users’ thinking during interviews

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

What co-speech gestures reveal about users’ thinking during interviews

Friday, June 30, 2023 • Rosenfeld Community
Share the love for this talk
What co-speech gestures reveal about users’ thinking during interviews
Speakers: Dane DeSutter and Stephanie Scopelitis
Link:

Summary

Drs. DeSutter and Scopelitis discussed how User Experience (UX) researchers can triangulate and enrich information from one-on-one interviews by attending to users’ co-speech gestures—the spontaneous movements that humans make with their hands and body when communicating. Gestures are a “window to the mind” and can reveal unspoken information about users’ emotional states as well as the structure and composition of their mental models. They concluded with a practical guide for efficiently implementing gesture research.

Key Insights

  • Gestures provide a non-verbal window into users' mental models, often revealing thoughts not expressed in speech.

  • Representational gestures, especially those made in personal gesture space, indicate cognitive processes and implicit imagery.

  • Users commonly hold multiple, context-dependent mental models rather than a single static one.

  • In interviews, interviewer gestures increase participant gesturing and improve conversational rapport.

  • Video interviews pose challenges for capturing gestures fully; positioning and prompting can mitigate this.

  • Speech-gesture mismatches often signal ongoing mental model construction or word searching by users.

  • Gestures can reveal emotional attachment or disengagement with technology, influencing adoption and retention.

  • Mental models can be anchored by recent technology prototypes, such as chat GPT for AI understanding.

  • Structured interview protocols that elicit gesturing and separate talking from tool use optimize gesture data collection.

  • Open source motion tracking and gesture analysis tools can aid qualitative research by quantifying gesture patterns.

Notable Quotes

"Gestures are a window to the mind."

"Gesture and speech form an integrated system; they reinforce one another."

"We’re really leaving half of our data on the table by not attending to gesture when eliciting mental models."

"Gesture is not computer and smartphone gestures, but spontaneous movements people make with hands and arms."

"Four me gestures happen in that personal gesture space and serve as thinking tools for the speaker."

"When gestures and speech mismatch, it often means the speaker is still refining their mental model."

"Without looking at the gesture, we would have come to a less complete mental model."

"Users have more than one mental model; they can be constructed on the fly depending on context."

"The degree to which the user feels in control with an intelligent agent brings up conversational mental models."

"The more you gesture, the more your interviewee will gesture."

Ask the Rosenbot
Maria Giudice
Empowering change: Reigniting purpose, passion and impact in research
2025 • Advancing Research 2025
Gold
Sheryl Cababa
Day 2 Panel
2024 • Designing with AI 2024
Gold
Mariesa Lenz
What Beekeeping Taught me about Product Teams
2025 • Rosenfeld Community
Alnie Figueroa
Teamwork: Strategies for Effective Collaboration with Other Program Management Teams
2022 • DesignOps Summit 2022
Gold
Alexandra Schmidt
Why Ethics Can't Save Tech
2022 • Civic Design 2022
Gold
Steve Portigal
War Stories LIVE! Steve Portigal
2020 • Advancing Research 2020
Gold
Kristin Skinner
Theme 1 Intro
2021 • DesignOps Summit 2021
Gold
Chris Govias
Perspectives on Civic Design
2021 • Civic Design Community
George Abraham
Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)
2021 • DesignOps Summit 2021
Gold
Gabrielle Verderber
Documentation Your Team Will Actually Use
2023 • DesignOps Summit 2023
Gold
Dave Hora
A Research Skills Evolution
2021 • Advancing Research 2021
Gold
Steve Sanderson
Discussion
2015 • Enterprise UX 2015
Gold
Laureen Kattan
Centering Patients and Clinicians in a Complex Government Ecosystem
2023 • Design in Product 2023
Gold
Sheri Byrne-Haber
Accessibility at Scale
2021 • Design at Scale 2021
Gold
Ben Davies
Expert Panel: The Principles of Research Repository Design
2022 • Advancing Research 2022
Gold
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold

More Videos

Jeff Ephraim Bander

"You should always have your research questions first before deciding to use eye tracking."

Jeff Ephraim Bander Ariane Rahn Philipp Reiter

Eye Tracking Gamechanger: Why Smartphone Eye Tracking will Revolutionize Your UX Research

March 11, 2022

Marina Martin

"By getting all the people in one room — doctors and claims processors — we had them demonstrate their processes to each other. Minds blown."

Marina Martin

Lives on the Line: The Stakes of UX at the Scale of Government

June 14, 2018

Ariba Jahan

"Creating space to name what we’re experiencing without guilt allowed us to unload some of that emotional weight."

Ariba Jahan

Team Resiliency Through a Pandemic

January 8, 2024

Marisa Bernstein

"No surprises. Tell participants everything they’ll be experiencing from who will be in the room to if the session will be recorded."

Marisa Bernstein

It Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing

December 9, 2021

Mike Brzozowski

"Sustainability isn't binary but multi-attribute, and it varies over time and geography."

Mike Brzozowski Laura Palotie Steve Isley Nancy Tsang

UX in everyday products: Empowering climate conscious choices

July 17, 2024

Jon Fukuda

"Empathy means constantly going back to the people using the tools and hearing their feedback."

Jon Fukuda Amy Evans Ignacio Martinez Joe Meersman

The Big Question about Innovation: A Panel Discussion

September 25, 2024

Kate Kalcevich

"It's cheaper to fix accessibility issues in the design phase than to wait until after coding or production."

Kate Kalcevich

Integrating Accessibility in DesignOps

September 23, 2024

Nalini P. Kotamraju

"Transparency and honesty are the only ways I handle difficult questions about compromises for the greater good."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021

Amy Parness

"Commercial buyers evaluate products primarily on needs first; certifications like Energy Star are bonuses."

Amy Parness

Scaling Sustainability: Complementary strategies that drive long-term success

May 29, 2025