Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"There is not a book currently that is dedicated to design ops. We want this to be the definitive handbook."
John Calhoun Rachel PosmanBring your DesignOps Story to Life! The Definitive DesignOps Book Jam
October 3, 2023
"People in some countries are three times more likely to buy a product if it’s localized in their language."
Nancy DouyonWe'll Figure That Out in the Next Launch: Enterprise Tech's Nobility Complex
June 15, 2018
"If you think of research methods as design patterns, the interview is a basic design pattern at the heart of most qualitative research methods."
Veevi RosensteinBuilding for Scale: Creating the Zendesk UX Research Practice
January 8, 2024
"In women, ADHD can sometimes just be seen as being chatty or forgetful or clumsy, as opposed to restless and disruptive."
Jessica NorrisADHD: A DesignOps Superpower
September 9, 2022
"LLMs talk the talk but don’t understand the meaning. They’re master impersonators without true knowledge."
Josh Clark Veronika KindredSentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
January 15, 2025
"Directive Disruptors plow through organizations without listening, causing alienation and low morale."
Jason Mesut Martina Hodges-Schell Jose CoronadoUnmasking Design Leadership: Navigating leadership without neglecting ourselves
October 30, 2025
"Sharing war stories helps illustrate the necessity of flexibility and being present in the moment."
Steve Portigal Susan Simon-Daniels Tamara Hale Randolph Duke IIWar Stories LIVE! Q&A-Discussion
March 30, 2020
"Showing people that design Ops will make their lives easier is key to adoption."
Meredith Black Elyse HornbacherBuilding Community and Common Trends to Look for in 2021
December 17, 2020
"Acknowledging that you can’t get other people to do things is truly the path to freedom."
Matt LeMayYou Don’t “Get” Anyone to Do Anything
December 6, 2022
