Summary
You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"The System Usability Scale’s question about needing a technical person to use the system is confusing and inaccurate for screen reader users."
Sam ProulxSUS: A System Unusable for Twenty Percent of the Population
December 9, 2021
"It's kind of Peace Corps for nerds – people come in for a tour and try to have impact at scale."
Michael LandEstablishing Design Operations in Government
February 18, 2021
"VOC wasn’t a replacement for user research, but an enhancer that captured nuggets researchers might miss during discovery."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"Leadership doesn’t only mean managing; it can be mentoring, product leadership, or technical influence."
Ian SwinsonDesigning and Driving UX Careers
June 8, 2016
"The persona of a goalkeeper is helping to triage and prioritize strategy and work for the executive."
Isaac HeyveldExpand DesignOps Leadership as a Chief of Staff
September 8, 2022
"Change is messy and it can be uncomfortable, much like baking bread—it’s hard to imagine sticky dough turning into a perfect loaf."
Amy EvansHow to Create Change
September 25, 2024
"Teleportation lets us move throughout the organization by building tools that gather disparate data for real-time decision making."
Kate Koch Prateek KalliFlex Your Super Powers: When a Design Ops Team Scales to Power CX
September 30, 2021
"You lie to yourself more than anyone else; the lie is that you are completely rational and objective."
Dave GrayLiminal Thinking: Sense-making for systems in large organizations
May 14, 2015
"Product teams want quick, grab-and-go research results, but reusable insights are more abstract and take longer to digest."
Matt DuignanAtomizing Research: Trend or Trap
March 30, 2020
