Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?

Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Brendan Jarvis
Framing Tomorrow by Questioning Today
2022 • Design at Scale 2022
Gold
Jamika Burge
Embracing change: Navigating shifting landscapes with compassion and agency
2025 • Advancing Research 2025
Gold
Emily Eagle
Can't Rewind: Radio and Retail
2019 • Enterprise Experience 2019
Gold
Yasmine Khan
Checking Bias and Listening to Financially Vulnerable Americans
2020 • Advancing Research 2020
Gold
Adam Cutler
People + Places + Practices = Outcomes
2016 • Enterprise UX 2016
Gold
Dan Hill
Strategic design, slowdown, and the infrastructures of everyday life
2022 • Enterprise Community
Alëna Iouguina
Designing Systems at Scale
2018 • DesignOps Summit 2018
Gold
Bud Caddell
Theme 2 Intro
2021 • DesignOps Summit 2021
Gold
Amy Evans
How to Create Change
2024 • DesignOps Summit 2024
Gold
Ariel Kennan
Civic Design in 2022
2022 • Civic Design Community
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
John Paul de Guzman
10k Screens Later: How We Became a Data-Driven Design Organization
2024 • DesignOps Summit 2024
Gold
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Aditi Ruiz
A PM State of Mind: Empathy Mapping Your Product Manager, Pt. 1
2022 • Design in Product 2022
Gold
Rebecca Topps
Planning and conducting remote usability studies for accessibility
2020 • Advancing Research Community
Natalie Dunbar
DesignOps and Content Strategy: Envisioning the Future Together
2021 • DesignOps Summit 2021
Gold

More Videos

Victor Udoewa

"Songs become methodologies—the way you identify who is at the door is through the song they sing."

Victor Udoewa

Beyond Methods and Diversity: The Roots of Inclusion

March 26, 2024

Peter Morville

"When we understand organizations as ecosystems, we recognize the need for organic simplicity."

Peter Morville

The Architecture of Understanding

May 13, 2015

Sam Ladner

"The boss’s son is no longer the default backup boss; there are now written rules accessible to all."

Sam Ladner

How Research Can Drive Strategic Foresight

March 9, 2022

Michelle Bejian Lotia

"The feedback culture in our company means there’s transparent discussion if conclusions aren’t clear or agreed upon."

Michelle Bejian Lotia Anne-Marie Morell

Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization

March 28, 2023

Leisa Reichelt

"My kids know everything. If you listen to conversations with them, they say I know, Ma, I know."

Leisa Reichelt

Opening Keynote: Operating in Context

November 7, 2018

Sarah Gallimore

"Artifacts of the future help us envision and grasp the change that we’re hoping to have—it’s not just bullet points or vision statements."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Joerg Beringer

"The output is not a requirements statement but a description of what is happening in the context."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

September 10, 2025

Peter Merholz

"Design is always going to be subjective, so you have to figure out what quality looks like in your organization."

Peter Merholz

Customer-Centered Design Organizations

June 8, 2017

Dalia El-Shimy

"Closing the loop by understanding when and why recommendations don’t get adopted is probably the most easily actionable piece of advice I can give."

Dalia El-Shimy

So You've Got a Seat at the Table. Now What?

March 31, 2020