Are My Research Findings Actually Meaningful?
Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"I have learned not to hold on too tightly to the ideas I’ve constructed about myself."
Tamara HaleWar Stories LIVE! Tamara Hale
March 30, 2020
"Building strong relationships with marketing requires mutual respect and regularly following up on their insights."
Joanna Vodopivec Prabhas PokharelOne Research Team for All - Influence Without Authority
March 9, 2022
"We use storytelling to turn accessibility from just compliance into a business differentiator that executives care about."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"I fumed that my idea had been ignored, wondering if it was because I was new, a woman, brown, or had a PhD."
Nalini P. KotamrajuTwo Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"
March 10, 2021
"Simple word-list based filters can easily be tricked by misspellings or slight variations in prompts."
Ian JohnsonLatent Scope: Finding structure in unstructured data
June 11, 2025
"End your day by writing the to-do list for tomorrow; it’s easy to get distracted when you’re not in a full-time gig."
Greg PetroffExit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
September 24, 2025
"The word intelligence in artificial intelligence leads many to read more into the technology than is actually there."
Frances Yllana Jorge Arango Maria Taylor Briana ThomasThe Big Question about Impact: A Panel Discussion
September 24, 2024
"Organizational silos are killers."
Maria GiudiceRemaking the Making Company: Moving from Product to Experience
June 9, 2016
"Evergreen content needs a strong perspective on why the topic matters along with practical examples on how to apply it."
Louis RosenfeldCoffee with Lou: Should You Write a (UX) Book?
March 7, 2024
