Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"OKRs can sometimes suck the oxygen out of the room if they focus only on bottom-line results, missing process and experience."
Bria Alexander Benson Low Natalya Pemberton Stephanie GoldthorpeOKRs—Helpful or Harmful?
January 20, 2022
"Adobe’s Kickbox is a fantastic professional development tool, but it doesn’t drive new lines of business."
Jeff GothelfInnovation Studios: the Engines of Enterprise Experimentation
May 14, 2015
"Half of Airbnb’s traffic is from non-English devices and 80% of bookings involve at least one party outside the U.S."
Wyatt HaymanGlobal Research Panels
August 8, 2020
"We encourage teams to build simulated persona testing into regular rituals—daily, weekly, or quarterly—to continuously refine messaging."
Gillian Salerno-Rebic Mark MicheliFrom Insight to Impact: How JourneySpark Used WEVO Pulse + Pro to Drive a 50% Lift in Ad Engagement
June 11, 2025
"As a civic designer, it’s really up to you to hold managers accountable for their promises on inclusion."
Yalenka Mariën Marie MervaillieDesigning for Digital Inclusion in the Belgian Government
December 8, 2021
"Dark patterns like Airbnb's price concealment and Twitter's neglect of accessibility were designed decisions, whether designers were involved or not."
Angelos ArnisNavigating the Rapid Shifts in Tech's Turbulent Terrain
October 2, 2023
"It was like giving a map that you could not read or even understand."
Benjamin RealMaturity Models: A Core Tool for Creating a DesignOps Strategy
October 1, 2021
"We started the digital service as a subsidiary company during the pandemic and delivered first services within a year."
Magdalena ZadaraZero Hour: How to Get Far Quickly When Starting Your Digital Service Unit Late
November 16, 2022
"Designing for edge cases can lead to solutions everyone wants to use because they’re more comfortable and inclusive."
Billy CarlsonIdeation tips for Product Managers
December 6, 2022
Latest Books All books
Dig deeper with the Rosenbot
How can senior designers coach junior designers to maintain intentional design decisions while using AI?
How do product risk categories like feasibility, viability, desirability, and usability relate to research method choice?
What challenges does working under constraints at large tech companies create for UX designers?