Rosenverse
Experimental research: techniques for deep, psychology-driven insights

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Experimental research: techniques for deep, psychology-driven insights

Gold
Wednesday, March 12, 2025 • Advancing Research 2025
Share the love for this talk
Experimental research: techniques for deep, psychology-driven insights
Speakers: Katie Hansen
Link:

Summary

In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.

Key Insights

  • •

    Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.

  • •

    Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.

  • •

    Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.

  • •

    Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.

  • •

    Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.

  • •

    Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.

  • •

    Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.

  • •

    Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.

  • •

    Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.

  • •

    Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.

Notable Quotes

"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."

"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."

"Randomization ensures differences between groups come from the variable tested, not external factors."

"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."

"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."

"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."

"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."

"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."

"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."

"Start small with one clear testable hypothesis and build from there to make experimental research approachable."

Ask the Rosenbot
Feyikemi Akinwolemiwa
Uninvited, Not Unwelcome: What Researchers Bring When Nobody Asked
2026 • Rosenfeld Community
Laura Schaefer
DesignOps: A Conduit for Inclusion
2022 • DesignOps Summit 2022
Gold
Katy Mogal
But Do Your Insights Scale?
2021 • Advancing Research 2021
Gold
Jaskiran Kang
Why Community is Key to Professionalizing Design
2022 • Civic Design Community
Dan Willis
Enterprise Storytelling Sessions
2016 • Enterprise UX 2016
Gold
Rachel Posman
"Ask Me Anything" with Rachel Posman and John Calhoun, Authors of the Upcoming Rosenfeld Book, The Design Conductors
2024 • DesignOps Summit 2024
Gold
Rachael Dietkus, LCSW
Trauma-Responsive Design: Reimagining the Future of Design Now
2021 • Civic Design 2021
Gold
Bria Alexander
OKRs—Helpful or Harmful?
2022 • DesignOps Community
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps Summit 2024
Gold
Deanna Zandt
The Unspoken Complexity of “Self-Care” with Deanna Zandt
2022 • Civic Design Community
Vasileios Xanthopoulos
A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale
2024 • Enterprise Experience 2020
Gold
Weidan Li
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
2024 • Designing with AI 2024
Gold
Jennifer Kanyamibwa
Creating the Blueprint: Growing and Building Design Teams
2018 • DesignOps Summit 2018
Gold
Jen Briselli
Learning Is The Engine: Designing & Adapting in a World We Can’t Predict
2025 • Rosenfeld Community
Josh Clark
Sentient Design: Crafting Intelligent Interfaces with AI
2026 • Designing with AI 2026
Gold
Joe Meersman
Use AI to Drive Outcomes that Go Beyond the Design Sprint
2024 • DesignOps Summit 2024
Gold

More Videos

Carla Casariego

"When we started, user-centered design was a relatively new concept to the organization."

Carla Casariego Sarah Spencer

DesignOps in Wonderland

October 24, 2019

Christian Crumlish

"Product manager is not the CEO of the product and I don’t think that’s a very helpful metaphor."

Christian Crumlish

AMA with Christian Crumlish, author of Product Management for UX People

March 24, 2022

Samuel Proulx

"If you think about the baby boomer generation, they are getting older, but their reliance on technology is not decreasing."

Samuel Proulx

Designing for Disability, Innovating for Everyone

March 11, 2025

Kate Towsey

"You've got to slow down to go fast; make space in your schedule to think strategically."

Kate Towsey

Ask Me Anything (AMA) with Kate Towsey

April 2, 2025

Sam Proulx

"Alternative navigation includes things like switch systems, voice dictation, and head mice to replace standard keyboard and mouse."

Sam Proulx

SUS: A System Unusable for Twenty Percent of the Population

September 29, 2021

Jim Kalbach

"Jazz soloists draw from a lifetime of patterns, sometimes quoting TV theme songs like the Muppets or Sanford and Son."

Jim Kalbach

Jazz Improvisation as a Model for Team Collaboration

November 6, 2017

Jorge Arango

"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."

Jorge Arango

[Demo] How to re-categorize content at scale using LLMs

June 5, 2024

Erin Weigel

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

Erin Weigel

UX Lessons from running more than 1,200 A/B Tests

July 10, 2024

Indi Young

"Talking about thinking styles is just as easy and fast a shortcut as talking about demographics, but more useful."

Indi Young

Thinking styles: Mend hidden cracks in your market

January 8, 2025

Latest Books All books