Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?

Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Roberta Dombrowski
Making Research a Team Sport
2022 • Advancing Research 2022
Gold
Megan Kierstead
You Are a Badass at UX: Overcoming Imposter Syndrome
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
George Abraham
Design Systems To-Go: Reimagining Developer Handoff, and Introducing App Builder (Part 2)
2021 • DesignOps Summit 2021
Gold
Harry Brignull
Beyond Clicks and Tricks: Why deceptive design has grown into a regulatory faultline
2026 • Rosenfeld Community
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Civic Design 2022
Gold
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Gold
Bria Alexander
Theme Two Intro
2022 • DesignOps Summit 2022
Gold
Rachael Dietkus, LCSW
Leading through the long tail of trauma
2022 • Enterprise Community
Lija Hogan
Doing more with more: Lessons from the Front Lines of Democratization
2022 • Advancing Research 2022
Gold
Kristin Skinner
Theme 1 Intro
2021 • DesignOps Summit 2021
Gold
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Greg Nudelman
Designing Conversational Interfaces
2019 • Enterprise Community
Jorge Arango
AI as Thought Partner: How to Use LLMs to Transform Your Notes (3rd of 3 seminars)
2024 • Rosenfeld Community
Sohit Karol
Designing Delightful Listening Experiences: Mixed Methods Research in the Age of Machine Learning
2020 • Advancing Research 2020
Gold
Dave Hora
Research in the Face of Complexity: New Sensibility for New Situations
2025 • Rosenfeld Community

More Videos

Tamara Hale

"I have learned not to hold on too tightly to the ideas I’ve constructed about myself."

Tamara Hale

War Stories LIVE! Tamara Hale

March 30, 2020

Joanna Vodopivec

"Building strong relationships with marketing requires mutual respect and regularly following up on their insights."

Joanna Vodopivec Prabhas Pokharel

One Research Team for All - Influence Without Authority

March 9, 2022

Saara Kamppari-Miller

"We use storytelling to turn accessibility from just compliance into a business differentiator that executives care about."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Nalini P. Kotamraju

"I fumed that my idea had been ignored, wondering if it was because I was new, a woman, brown, or had a PhD."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021

Ian Johnson

"Simple word-list based filters can easily be tricked by misspellings or slight variations in prompts."

Ian Johnson

Latent Scope: Finding structure in unstructured data

June 11, 2025

Greg Petroff

"End your day by writing the to-do list for tomorrow; it’s easy to get distracted when you’re not in a full-time gig."

Greg Petroff

Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian

September 24, 2025

Frances Yllana

"The word intelligence in artificial intelligence leads many to read more into the technology than is actually there."

Frances Yllana Jorge Arango Maria Taylor Briana Thomas

The Big Question about Impact: A Panel Discussion

September 24, 2024

Maria Giudice

"Organizational silos are killers."

Maria Giudice

Remaking the Making Company: Moving from Product to Experience

June 9, 2016

Louis Rosenfeld

"Evergreen content needs a strong perspective on why the topic matters along with practical examples on how to apply it."

Louis Rosenfeld

Coffee with Lou: Should You Write a (UX) Book?

March 7, 2024