Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Dig deeper—ask the Rosenbot:
More Videos
"One eye on the horizon and the other on the next wave is the perspective teams need to deliver quality digital products."
Maria SkaadenContinuous Design: One eye on the horizon and the other on the next wave
November 8, 2018
"Anytime you have to justify something by saying just this one time, you know you’re going down a bad road."
Dan WillisEnterprise Storytelling Sessions
May 13, 2015
"Even though we're virtual, our code of conduct still applies and requires respect and kindness."
Bria AlexanderOpening Remarks
January 8, 2024
"The store mobile device is the indispensable silent partner enabling store employees to balance task productivity and customer engagement."
Catherine DubutBridging Physical and Digital Spaces: Approaches to Retail Service Design
March 18, 2021
"No one is married to a company; people will leave if their needs aren’t met, so retention and team happiness are important success indicators."
Angelos ArnisState of DesignOps: Learnings from the 2021 Global Report
October 1, 2021
"Switching sideways from management to principal IC was a leap of faith but ultimately gave me more impact."
Edward CuppsThe Principal Path: Journeying from Management to Individual Contributor
June 11, 2021
"Diagrams help when we feel stuck—providing stability, transparency, understanding, clarity, and kindness."
Abby CovertStuck? Diagrams Help
October 27, 2022
"We protect the time and headspace of everyone on the design team so they can focus more on their craft."
Jilanna WilsonDistributed DesignOps Management
February 26, 2019
"What doesn’t kill you will only make you stronger."
Maria GiudiceEmpowering change: Reigniting purpose, passion and impact in research
March 13, 2025
