Experimental research: techniques for deep, psychology-driven insights
Summary
Key Insights
-
•
Experimental research goes beyond AB testing by revealing why changes work, revealing causation instead of just correlation.
-
•
Testing 20 trust signals simultaneously in one experiment is far more resource-efficient than running 20 separate AB tests.
-
•
Trust is multi-dimensional; measuring it at both platform and individual pro levels offers a more holistic understanding.
-
•
Random assignment and control groups are essential for isolating effects and attributing results to manipulated variables.
-
•
Half of the 20 trust signals tested had statistically significant positive effects on overall trust in Thumbtack.
-
•
Ten trust signals demonstrated broad impact across all eight trust metrics, providing a clear prioritization roadmap for product teams.
-
•
Experimental research can help product teams prioritize smartly by focusing on hypotheses likely to succeed before costly AB tests or full launches.
-
•
Ethics in experimentation require transparency, informed consent, and minimization of harm to participants.
-
•
Hypothetical scenarios in experiments improve control but have limitations in ecological validity, which can be addressed by validating in live products.
-
•
Understanding that correlation does not equal causation is crucial for UX researchers to avoid misleading conclusions and to rely on experiments.
Notable Quotes
"When we hear experiment, most think AB testing, but experimental research goes beyond to uncover the why behind outcomes."
"The independent variable is what you manipulate, dependent variable is what you measure to see if there’s an effect."
"Randomization ensures differences between groups come from the variable tested, not external factors."
"Half of the trust signals had a statistically significant positive effect on overall trust in Thumbtack, which is a big deal."
"Testing 20 trust signals simultaneously saved significant time and resources compared to running individual AB tests."
"Trust is complex, so we measured eight different trust-related outcomes at both the platform and professional level."
"Ethics are foundational in experiments since manipulating variables can impact participants' perceptions and emotions."
"Correlation does not equal causation. To say X causes Y, you’ve gotta do an experiment."
"Experimental research bridges the gap between curiosity and action by turning ideas into evidence and evidence into impact."
"Start small with one clear testable hypothesis and build from there to make experimental research approachable."
Or choose a question:
More Videos
"When we started, user-centered design was a relatively new concept to the organization."
Carla Casariego Sarah SpencerDesignOps in Wonderland
October 24, 2019
"Product manager is not the CEO of the product and I don’t think that’s a very helpful metaphor."
Christian CrumlishAMA with Christian Crumlish, author of Product Management for UX People
March 24, 2022
"If you think about the baby boomer generation, they are getting older, but their reliance on technology is not decreasing."
Samuel ProulxDesigning for Disability, Innovating for Everyone
March 11, 2025
"You've got to slow down to go fast; make space in your schedule to think strategically."
Kate TowseyAsk Me Anything (AMA) with Kate Towsey
April 2, 2025
"Alternative navigation includes things like switch systems, voice dictation, and head mice to replace standard keyboard and mouse."
Sam ProulxSUS: A System Unusable for Twenty Percent of the Population
September 29, 2021
"Jazz soloists draw from a lifetime of patterns, sometimes quoting TV theme songs like the Muppets or Sanford and Son."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
November 6, 2017
"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."
Jorge Arango[Demo] How to re-categorize content at scale using LLMs
June 5, 2024
"No tracking is perfect because numbers tell you what likely happened, but not why it happened."
Erin WeigelUX Lessons from running more than 1,200 A/B Tests
July 10, 2024
"Talking about thinking styles is just as easy and fast a shortcut as talking about demographics, but more useful."
Indi YoungThinking styles: Mend hidden cracks in your market
January 8, 2025