Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Experimental research: techniques for deep, psychology-driven insights

Gold
Wednesday, March 12, 2025 • Advancing Research 2025
Share the love for this talk
Experimental research: techniques for deep, psychology-driven insights
Speakers: Katie Hansen
Link:

Summary

In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.

Key Insights

  • Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.

  • Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.

  • Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.

  • Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.

  • Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.

  • Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.

  • Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.

  • Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.

  • Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.

  • Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.

Notable Quotes

"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."

"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."

"Randomization ensures differences between groups come from the variable tested, not external factors."

"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."

"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."

"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."

"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."

"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."

"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."

"Start small with one clear testable hypothesis and build from there to make experimental research approachable."

Ask the Rosenbot
Kristin Sundermeyer
Design Ops Metrics
2021 • DesignOps Summit 2021
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Bob Baxley
Leading with Design Operations Past and Present
2019 • DesignOps Community
Verónica Urzúa
The B-side of the Research Impact
2021 • Advancing Research 2021
Gold
Maria Rosala
AI-Assisted Qualitative Analysis: What to Automate and What to Own
2026 • Rosenfeld Community
Billy Carlson
Pro-level UI Tips for Beginners
2022 • DesignOps Summit 2022
Gold
Dr. Nikki Smith
Research Strategy: Connecting Insights to Outcomes
2025 • Advancing Research 2025
Gold
Todd Healy
Driving Change with CX Metrics
2023 • Enterprise UX 2023
Gold
Robert Fabricant
Industry junctures: Paths forwards for UXR and the critical decisions that get us there [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Jorge Arango
The Best of Both Worlds: How to Integrate Paper and Digital Notes (1st of 3 seminars)
2024 • Rosenfeld Community
Bria Alexander
Opening Remarks
2023 • Advancing Research 2023
Gold
Sha Hwang
The First Fifty Years of Civic Design
2022 • Civic Design 2022
Gold
Dr. Jamika D. Burge
A Genuine Conversation about the Future of UX Research
2024 • Advancing Research Community
Kyle Godbey
Non-linear service design for complex adaptive systems
2025 • Rosenfeld Community
Marc Rettig
The Urge to Migrate
2026 • Rosenfeld Community
Noah Bond
Redefining truth and inclusivity: Navigating data ownership and ethical research in the age of disinformation
2025 • Advancing Research 2025
Gold

More Videos

Wyatt Hayman

"We send out surveys weekly, but respondents have two weeks to answer, accommodating different time zones and bandwidth."

Wyatt Hayman

Global Research Panels

August 8, 2020

Ashley Cortez

"Community-centered design flips the traditional script to give design power to those most impacted by policies and services."

Ashley Cortez Devika Menon Nidhi Singh Rathore Danita J. Reese Mari Nakano

Shifting Toward Community-Led Innovation in Local Government

December 8, 2021

Shahrzad Samadzadeh

"Some situations are not worth saving. If you have no champion, no clarity, and no purpose, is it worth it to continue?"

Shahrzad Samadzadeh

What Is My Value? Two Takes and Some Mistakes

January 8, 2024

Lily Aduana

"When working with an agency, the quoting process takes about two weeks before recruitment even starts."

Lily Aduana Savannah Hobbs Brittany Rutherford

5 Reasons to Bring Your Recruiting in-House (and How To Do It)

March 12, 2021

Tim Parmee

"We’ve honed and refined our mechanisms for output, for scale, for growth, and even in moments of high uncertainty, organizations still measure success through quantity."

Tim Parmee

Changing Our Design Pressure Points

October 2, 2023

Adam Thomas

"Being data informed is not data driven—blindly following data can kill you, like the Challenger explosion."

Adam Thomas

Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way

December 6, 2022

Doug Powell

"In 2023, design leaders were being restructured out and design relegated to a tactical, downstream function."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Robert Schwartz

"In order to do this kind of work, you have to be willing to take a bloody nose. Those bloody noses are badges of honor."

Robert Schwartz

We're Here for the Humans

June 9, 2017

Rima Campbell

"Numbers drive conversations, numbers drive decisions. That is the executive language."

Rima Campbell Amrit S Bhachu

Increase Productivity and Drive Business Impact

September 24, 2024