Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"Bringing insights gathering functions together helps us deliver data in a thoughtful way."
Brad Peters Anne MamaghaniShort Take #1: UX/Product Lessons from Your Industry Peers
December 6, 2022
"Creating an environment where people feel safe to try design helps build enthusiasm and psychological safety."
Lona MooreScaling Design Beyond Designers
June 11, 2021
"We can’t design for every possible outcome anymore. We have to anticipate fuzzy ranges of results and channel behavior accordingly."
Josh Clark Veronika KindredSentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
January 15, 2025
"If we can’t provide research as a service at scale, we need to make sure the teams are equipped to do it on their own."
Erin May Roberta Dombrowski Laura Oxenfeld Brooke HintonDistributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
March 10, 2022
"People were more likely to explain their thought process and just more context around the particular situation of their org."
Tara TresselInvestigating qualitative depth of AI-moderated interviews
March 10, 2026
"Kate said she considers live code to be the source of truth because she can't trust the design site due to inconsistencies."
Charles Lee Jennie YipBuilding a New Home for the Atlassian Design System
October 22, 2020
"When we tailor education to individual needs, everyone benefits."
Kristin SkinnerFive Years of DesignOps
September 29, 2021
"Bringing in expert interviews and secondary data gives you an outsider’s perspective to challenge the status quo."
Megan BlockerGetting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
March 26, 2024
"Shared mental models serve as organizing frameworks that all players in the system can understand and make decisions within."
Shanti Mathew Natalie Sims Natalia RadywylCivic Design at Scale: Introducing the Public Policy Layer Cake
December 9, 2021
