Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Sharbani Dhar
Breathing Room for Delight
2024 • Enterprise Experience 2020
Gold
Alla Weinberg
Workers Are Sick of Change: The Cure is Psychological Safety
2023 • Enterprise UX 2023
Gold
Steve Baty
Discussion
2016 • Enterprise UX 2016
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Smitha Papolu
Theme 3 Discussion
2024 • Enterprise Experience 2020
Gold
Alnie Figueroa
Teamwork: Strategies for Effective Collaboration with Other Program Management Teams
2022 • DesignOps Summit 2022
Gold
Kristen Guth, Ph.D.
Out of the FOG: A Non-traditional Research Approach to Alignment
2023 • Advancing Research 2023
Gold
Ryan Matthew
Bridging Design and Code: AI-Powered Design System Integration
2025 • DesignOps Summit 2025
Gold
Gabriela Barneva
Operationalizing Inclusive Design in Service Design
2025 • Advancing Service Design 2025
Conference
Jilanna Wilson
Distributed DesignOps Management
2019 • DesignOps Community
Ron Bronson
Design, Consequences & Everyday Life
2022 • Civic Design 2022
Gold
Dan Willis
Enterprise Storytelling Sessions
2019 • Enterprise Experience 2019
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • Designing with AI 2025
Gold
Dan Saffer
Why AI projects fail (and what we can do about it)
2025 • Rosenfeld Community
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Design at Scale 2022
Gold
Ned Dwyer
Right horses for the right courses – how and when to democratize research
2025 • Advancing Service Design 2025
Conference

More Videos

Doug Powell

"Diversity is beyond just the right thing to do. Our teams must reflect the world we’re designing for."

Doug Powell

Closing Keynote: Design at Scale

November 8, 2018

Mila Kuznetsova

"A growth mindset in research means being ready to adapt protocols based on what actually works with participants."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Sarah Gallimore

"We created a Wizard of Oz smoke and mirrors prototype—a simple text message thread between a parent and this service—just to get enough buy-in."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Lada Gorlenko

"If your users are happy, that anecdotal feedback often carries more weight early than quantitative metrics."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Alnie Figueroa

"Think of AI as an additional team member pushing your creative thinking and expanding your idea range."

Alnie Figueroa

The Future of Design Operations: Transforming Our Craft

September 10, 2025

Landon Barnes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Emily Eagle

"Truly listening can push us beyond empathy to respect and reflection."

Emily Eagle

Can't Rewind: Radio and Retail

June 3, 2019

Malini Rao

"We neglected our power users during the responsive, one-size-fits-all strategy, and it really hurt us."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Asia Hoe

"I learned to be relentless about prioritization and to hone in on specific details and use cases."

Asia Hoe

Partnering with Product: A Journey from Junior to Senior Design

November 29, 2023