Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Sarah Fathallah
A Typology of Participation in Participatory Research
2023 • Advancing Research 2023
Gold
Gillian Salerno-Rebic
From Insight to Impact: How JourneySpark Used WEVO Pulse + Pro to Drive a 50% Lift in Ad Engagement
2025 • Designing with AI 2025
Gold
Louis Rosenfeld
Welcome / Housekeeping
2023 • Enterprise UX 2023
Gold
Theresa Marwah
How Atlassian is Operationalizing Respect in Research
2020 • Advancing Research Community
Jack Moffett
SAFe or Sorry?
2019 • Enterprise Community
Uday Gajendar
Day 1 Welcome
2024 • Designing with AI 2024
Gold
Josh Clark
Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
2025 • Rosenfeld Community
Ali Jeffery
How DesignOps Helped Enable Wall Street to Work Remotely
2020 • DesignOps Summit 2020
Gold
Xenia Adjoubei
Empowering Communities Through the Researcher in Residence Program
2023 • Advancing Research 2023
Gold
Meaghan Waters
Lack of Product Thinking will Doom Your Legacy Modernization
2021 • Design at Scale 2021
Gold
Mark Boulton
Ops without Designers
2018 • DesignOps Summit 2018
Gold
Louis Rosenfeld
Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?
2024 • Rosenfeld Community
Jake Burghardt
Stop wasting research: Unlock more value from research insights
2025 • Rosenfeld Community
Amy Jiménez Márquez
The Atypical UX Manager Path
2020 • Enterprise Community
Jess Greco
Claiming your power: Practical tools for amplifying your unique voice
2025 • Advancing Research 2025
Gold
Sarah Gallimore
Inspire Progress with Artifacts from the Future
2022 • Civic Design 2022
Gold

More Videos

Sam Proulx

"I’ve nearly bought products in my sleep because I memorized the key presses from consistent checkout flows."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023

Ignacio Martinez

"We want to build consultants who can solve problems across a variety of projects and clients."

Ignacio Martinez

Fair and Effective Designer Evaluation

September 25, 2024

Sarah Kinkade

"Using program dollars to hire freelancers helped me avoid difficult conversations with HR and finance early on."

Sarah Kinkade Mariana Ortiz-Reyes

Design Management Models in the Face of Transformation

June 8, 2022

Daniel Gloyd

"The Shaker village sidewalks followed people’s natural inclinations, not forcing unnatural movement."

Daniel Gloyd

Warming the User Experience: Lessons from America's first and most radical human-centered designers

May 9, 2024

Patrick Boehler

"Traditional metrics create a dangerous illusion where we optimize for what we can measure, not what people actually need."

Patrick Boehler

Fishing for Real Needs: Reimagining Journalism Needs with AI

June 10, 2025

Andy Barraclough

"You can chat with large sets of data in real time, not waiting hours or days for answers."

Andy Barraclough Betsy Nelson

From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme

September 23, 2024

Alexandra Schmidt

"I’m totally freaked out by facial recognition and wonder why we don’t get together as consumers to push back on bad designs."

Alexandra Schmidt

Why Ethics Can't Save Tech

November 18, 2022

Louis Rosenfeld

"In government, the motivation is reduction of misery: why are we up at 3 AM fixing something avoidable?"

Louis Rosenfeld

Discussion: What Operations can teach DesignOps

November 6, 2017

Mitchell Bernstein

"Carbon is just the starting point for what can be imagined. We're hired for our expertise to adapt, not just follow directions."

Mitchell Bernstein

Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape

September 29, 2021