Experimental research: techniques for deep, psychology-driven insights
Summary
In this session, we’ll unlock the transformative power of experimental research methods, inspired by best practices in the social sciences. We’ll explore what these techniques are and how they can be used to reveal powerful insights, drive innovation. You’ll also learn how to get started, how to apply them in ethical and rigorous ways, and why these methods are essential for deep, psychology-driven insights.
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"Participants want to take ownership of their stories and demand participatory, non-extractive, trauma-informed research methods."
Jemma Ahmed Megan Blocker Eduardo OrtizRedefining the research toolkit: Expanding methodologies for a changing world
March 12, 2025
"Every decision in a design system is like a puzzle piece; together they form a bigger picture that evolves over time."
Mitchell BernsteinOrganizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
September 29, 2021
"WMBE is a certification that requires paperwork but helps bring more diversity to contracting."
Emily LessardRFPs Without Tears: Writing Inclusive RFPS that Don't Scare Away Talent
December 9, 2021
"Executive sponsors accelerated adoption and signaled the value of the transformation effort across the organization."
Jenny PriceFrom Tradition to Transformation: Unlocking Startup Agility in a Legacy Enterprise
September 10, 2025
"Civic action as proof of concept to get government moving, to get the wheels turning was really interesting."
Sarah Auslander Betsy Ramaccia Gordon RossInsights Panel
November 18, 2022
"Large language models are where computers and language become indistinguishable, breaking down boundaries between cognition and technology."
Soma Ghosh Savina Hawkins Dave Hoffer Rob Thomas Victor UdoewaWhat emerging methods are advancing UX research [Advancing Research Community Workshop Series]
September 28, 2023
"Kate said she considers live code to be the source of truth because she can't trust the design site due to inconsistencies."
Charles Lee Jennie YipBuilding a New Home for the Atlassian Design System
October 22, 2020
"Our ways of being are roots that grow into ways of knowing, doing, and methods, all nourished by systems of value."
Victor UdoewaBeyond Methods and Diversity: The Roots of Inclusion
March 26, 2024
"There’s this kind of Collective unconscious within teams that formed over the pandemic which is deeper than any other team experience I’ve had."
DesignOps and The Great Talent War of 2021
August 19, 2021
