Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Are My Research Findings Actually Meaningful?
Gold
Thursday, March 10, 2022 • Advancing Research 2022
Share the love for this talk
Are My Research Findings Actually Meaningful?
Speakers: Landon Barnes
Link:

Summary

You should not be doing research for the sake of doing research. Research takes time and needs to be well throughout. More importantly, you need to determine if your findings are actually meaningful to the organization. In this session we will look at the idea of statistical significance and meaningfulness when reporting research findings.

Key Insights

  • Research must be tied to business imperatives and KPIs to deliver value, not done arbitrarily.

  • Qualitative research is crucial to interpret quantitative metrics like NPS meaningfully.

  • Statistical significance depends on sample size, variability, and magnitude of change.

  • A statistically significant change may not always be meaningful to the business context.

  • Meaningfulness is determined by alignment with organizational goals and market conditions.

  • Increasing sample size alone can make every change appear significant, which is misleading.

  • Driver modeling helps prioritize initiatives by linking key metrics to business outcomes.

  • Benchmarking against competitors and cross-industry experiences anchors understanding of CX scores.

  • Effective communication involves framing findings differently for executives, managers, and individual contributors.

  • Frequent measurement without changes made leads to wasted effort and misinterpretation.

Notable Quotes

"Research is not as easy as putting on a shoe and just doing it; it requires careful consideration and alignment with imperatives."

"Every board member can interpret an NPS score differently depending on their role and responsibilities."

"Statistical significance tells you if a change is likely not random, but it does not tell you if the change matters to your business."

"If you measure an infinite number of customers, every change would be statistically significant, but not every change would be meaningful."

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

"Driver modeling tells you which research initiatives matter most based on their impact on key outcome metrics."

"Customers benchmark your company to their last best experience, often outside of your industry."

"When executives start questioning your research details, that’s a good sign they’re engaged and trusting the process."

"You shouldn’t send another survey unless you’ve implemented changes and allowed time for behavior to adapt."

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Ask the Rosenbot
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • DesignOps Summit 2022
Gold
Alan Williams
Designing essential financial services for those in need
2022 • Civic Design Community
Mitchell Bernstein
Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
2021 • DesignOps Summit 2021
Gold
Tom Armitage
Day 2 Panel: Looking ahead: Designing with AI in 2026
2025 • Designing with AI 2025
Gold
Michelle Bejian Lotia
Rolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
2023 • Advancing Research 2023
Gold
Stephen Pollard
Closing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?
2017 • DesignOps Summit 2017
Gold
Smitha Papolu
Theme 3 Discussion
2024 • Enterprise Experience 2020
Gold
Uday Gajendar
Theme One Intro
2023 • Enterprise UX 2023
Gold
Janaki Kumar
Innovate with Purpose
2018 • Enterprise Experience 2018
Gold
Jemma Ahmed
Theme Panel
2025 • Advancing Research 2025
Gold
Maria Skaaden
Panel Discussion: Methodologies and Work Environments
2018 • DesignOps Summit 2018
Gold
Randolph Duke II
War Stories LIVE! Randy Duke II
2020 • Advancing Research 2020
Gold
Ilana Lipsett
Anticipating Risk, Regulating Tech: A Playbook for Ethical Technology Governance
2021 • Civic Design 2021
Gold
Kate Koch
Flex Your Super Powers: When a Design Ops Team Scales to Power CX
2021 • DesignOps Summit 2021
Gold
Dr. Nikki Smith
Research Strategy: Connecting Insights to Outcomes
2025 • Advancing Research 2025
Gold
Jackie Ajoux
Leveling-Up: A Single-Player’s Guide to the DesignOps Team-of-One
2024 • DesignOps Summit 2020
Gold

More Videos

Ana Ferreira

"We document everything from product specs to implementation and design specs to prevent knowledge silos."

Ana Ferreira

Designing Distributed: Leading Doist’s Fully Remote Design Team in Six Countries

January 8, 2024

Roy Opata Olende

"We automate the entire flow so once people react in Slack, they’re invited automatically to sessions."

Roy Opata Olende

How Zapier Uses ‘All Hands Research’ to Increase Exposure to Users

August 6, 2020

Kit Unger

"Service design is very up-and-coming, but there’s still a long way to go in thinking about the entire user journey."

Kit Unger Jackie Ho Veevi Rosenstein Vasileios Xanthopoulos

Theme 2: Discussion

January 8, 2024

Abbey Smalley

"Not growing the team was disheartening but led me to create training for partners to start taking a user-centered approach themselves."

Abbey Smalley Sylas Souza

Scaling UX Past the Size of Your Team

January 8, 2024

Helen Armstrong

"Building skepticism into users is essential because if you’re not skeptical as a designer, it’s hard to build it into your customers."

Helen Armstrong

Augment the Human. Interrogate the System.

June 7, 2023

Bas Raijmakers, PhD (RCA)

"We answer questions differently when we’re living life and doing activities than when we’re sitting down facing the camera."

Bas Raijmakers, PhD (RCA) Charley Scull Prabhas Pokharel

What Design Research can Learn from Documentary Filmmaking

March 11, 2022

Mila Kuznetsova

"Power dynamics exist in every session. People don’t want to be embarrassed or feel put on the spot."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Weidan Li

"Using analogy is very difficult for AI because it requires a unique human ability to connect and engage through storytelling."

Weidan Li

Qualitative synthesis with ChatGPT: Better or worse than human intelligence?

June 4, 2024

Christian Crumlish

"Leadership doesn’t necessarily mean managing people; it means scaling impact through teaching, training, and setting examples."

Christian Crumlish

AMA with Christian Crumlish, author of Product Management for UX People

March 24, 2022