Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Alana Washington
(Remote) Service Design: A Transformation Case Study
2022 • Design at Scale 2022
Gold
Victor Lombardi
Bridging Design and Climate Science
2024 • Climate UX Interest Group
Aiyana Bodi
Three Key Climate Initiatives and How You Can Help
2024 • Climate UX Interest Group
David Conrad
The Feeling of Data
2023 • Enterprise Community
Dan Hill
Designing for the infrastructures of everyday life
2024 • Designing with AI 2024
Gold
Prayag Narula
Empowering Designers to do Good Research
2022 • Advancing Research 2022
Gold
Brigette Metzler
Scaling ResearchOps: Helping Researchers do Their Best Work
2020 • Advancing Research 2020
Gold
Smitha Papolu
Theme 3 Discussion
2019 • Enterprise Experience 2019
Gold
Laura Weiss
Turn Down the Heat: 3 Ways to Handle Conflict in the Moment
2024 • Rosenfeld Community
John Mortimer
Panel Discussion
2024 • Advancing Service Design 2024
Gold
Lija Hogan
Doing more with more: Lessons from the Front Lines of Democratization
2022 • Advancing Research 2022
Gold
Bria Alexander
Reflect and Chart Forward
2021 • Civic Design 2021
Gold
Robert Schwartz
We're Here for the Humans
2017 • Enterprise Experience 2017
Gold
Louis Rosenfeld
Welcome / Housekeeping
2023 • Enterprise UX 2023
Gold
Andrew Webster
Scaling Design Capability: How Involved Should You Be?
2021 • DesignOps Summit 2021
Gold
Melissa Tsang
From Insights to Action: Driving Business Values through DesignOps
2024 • DesignOps Summit 2020
Gold

More Videos

Nalini Kotamraju

"I believed we could do more — more for the company, customers, stakeholders, and ourselves if we had more agency."

Nalini Kotamraju

Research After UX

March 25, 2024

Dean Broadley

"Constraints can channel focus, but being stuck in them limits potential."

Dean Broadley

Not Black Enough to be White

January 8, 2024

Denise Jacobs

"It can be so tiring to be the only one constantly saying this has to be done differently."

Denise Jacobs Nancy Douyon Renee Reid Lisa Welchman

Interactive Keynote: Social Change by Design

January 8, 2024

Kim Fellman Cohen

"What does ‘best work’ mean? It’s different for everyone, so our solutions have to be flexible and meaningful."

Kim Fellman Cohen

Measuring the Designer Experience

October 23, 2019

George Aye

"Power is the ability to change another person’s reality."

George Aye

That Quiet Little Voice: When Design and Ethics Collide

November 16, 2022

Nathan Curtis

"You need to be the connector between people trying to drive decisions and others influencing them."

Nathan Curtis

Beyond the Toolkit: Spreading a System Across People & Products

June 9, 2016

Greg Petroff

"We focus on delivering high quality but also moving fast—being comfortable negotiating that tension is important."

Greg Petroff

The Compass Mission

March 10, 2021

Chloe Amos-Edkins

"If you don’t do local research before launching a product, you risk disastrous failure."

Chloe Amos-Edkins

A Cultural Approach: Research in the Context of Glocalisation

March 27, 2023

Mackenzie Cockram

"We brought all the user journeys together with UX designers, content designers, product owners — it wasn’t just analytics and research in isolation."

Mackenzie Cockram Sara Branco Cunha Ian Franklin

Integrating Qualitative and Quantitative Research from Discovery to Live

December 16, 2022