Summary
In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Dig deeper—ask the Rosenbot:
More Videos
"When participants interact with researchers of different backgrounds, it can deeply affect their responses and the authenticity of data."
Sharon Banh Dave Hora Marieke McCloskey Alicia ZhongReimagining research: What does the field need to grow? [Advancing Research Community Workshop Series]
October 16, 2024
"I couldn’t be happier to be part of this conference and to advance the practice together."
Dr. Jamika D. BurgeTheme 3 Intro
March 11, 2022
"All our components can be data bound so the developer can point to any data source like CMS or JSON to populate UI elements naturally."
George Abraham Stefan IvanovDesign Systems To-Go: Reimagining Developer Handoff, and Introducing App Builder (Part 2)
October 1, 2021
"I know I will fail, but I will fail in small increments rather than bet everything and fail big."
Bill ScottLean Engineering: Engineering for Learning and Experimentation in the Enterprise
May 14, 2015
"Design review is simply the act of designers collaborating to increase quality and craft."
Erika Kincaid Brenna Heaps Jessica TsukimuraConnecting the Dots: How to Foster Collaboration and Build a Strong Design Review Culture
June 8, 2022
"We must create spaces that allow nature to thrive within our cities."
Lena ShenkarenkoCollaborative Wireframing for Creating Team Alignment and Shipping Better Products
October 21, 2020
"When we empower others, we are actually reinforcing the hierarchy we seek to subvert."
Victor UdoewaRadical Participatory Research: Decolonizing Participatory Processes
March 9, 2022
"The culture of business education has taught MBAs to be confident about many things that are simply not true."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020
"We can make this thing go faster, more people can access it, but we can have benefits for the planet as well."
James ChudleyDecarbonising User Journeys: How minimising enables us to do more with less
February 19, 2025
