Are My Research Findings Actually Meaningful?
Summary
Key Insights
-
•
Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.
-
•
Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.
-
•
Statistical significance depends on sample size, variability, and magnitude of change.
-
•
A statistically significant change may not always be meaningful to the business context.
-
•
Meaningfulness is determined by alignment with organizational goals and market conditions.
-
•
Increasing sample size alone can make every change appear significant, which is misleading.
-
•
Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.
-
•
Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.
-
•
Effective communication involves framing findings differently for executives, managers, and individual contributors.
-
•
Frequent measurement without changes made leads to wasted effort and misinterpretation.
Notable Quotes
"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."
"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."
"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."
"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."
"Customers benchmark your company to their last best experience, often outside of your industry."
"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."
"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."
"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."
Or choose a question:
More Videos
"Always remember the power of your values in times of doubt and change; lean into what your team really stands for."
Kim Holt Emma Wylds Pearl Koppenhaver Maisee XiongA Salesforce Panel Discussion on Values-Driven DesignOps
September 8, 2022
"If you feel uncomfortable or unsafe engaging in this community, we want to know about it."
Bria AlexanderOpening Remarks
March 29, 2023
"It’s about helping your colleagues be more productive and make better decisions to help customers."
Sofia QuinteroBeyond Tools: The Messy Business of Implementing Research Repositories
March 10, 2022
"Teams who translate complex goals into everyone’s language help people feel proud and aligned with broader business outcomes."
Christian Crumlish Aditi Ruiz Johanna Kollmann Catt SmallMorning Insights Panel
December 6, 2022
"If you have a question for a speaker, put it in the thread for their talk, not in the general chat."
Bria AlexanderOpening Remarks
November 18, 2022
"We are living in a world that for so many feels like it has been flipped on its head."
Jemma AhmedTheme 2 Intro
March 10, 2022
"If we had pulled something off the internet, half the time would be spent on critique."
Saara Kamppari-Miller"Prototype" vs "Prototype"--Breaking Down and Rebuilding Our Understanding of What We Do
October 24, 2019
"Knowledge management solutions must focus on both the explicit knowledge that can be documented and the tacit knowledge that resides in people’s experiences."
Maria TaylorKnowledge is Power: Managing the Lifeblood of the Design Org
October 3, 2023
"Difference-in-differences lets you study treatment effects in groups that start in very different places by looking at trends."
Joshua NobleCasual Inference
October 6, 2023
