Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
UX Lessons from running more than 1,200 A/B Tests
Summary
Key Insights
-
•
Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.
-
•
Execution often fails on good ideas rather than the ideas themselves.
-
•
Technical details like page load time can negate positive design changes if not carefully managed.
-
•
Edge case bugs in different languages or currencies can silently kill conversions.
-
•
Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.
-
•
Using system fonts improves legibility and page load speed, positively affecting conversions.
-
•
Tracking should only include users exposed to the tested change to avoid noisy data.
-
•
Qualitative user research drives many experiment ideas by revealing real user challenges.
-
•
Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.
-
•
Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.
Notable Quotes
"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."
"Compound effect means good upon good upon good eventually builds to incredibly fast growth."
"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."
"Increasing page load time by three seconds can nullify all your design improvements."
"Picking the wrong size image can impact customer experience as much as what you see on screen."
"There are as many versions of the website as stars in the sky because of language, currency, and device variations."
"No tracking is perfect because numbers tell you what likely happened, but not why it happened."
"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."
"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."
"You can’t fail 90 times out of 10 without laughing at yourself to keep going."
Or choose a question:
More Videos
"Our stakeholders wake up thinking about clinical outcomes, regulatory risk, reimbursement, not design principles."
Eric ShumakeDiagnosis UX: Building Influence in Healthcare Design
April 9, 2026
"Melinda Belcher has succeeded both as an outsider consultant and as an inside enterprise manager, and will share how to leverage those differences."
Dan WillisTheme 3: Intro
January 8, 2024
"Journey data allowed us to see that solution scale, not company size, better explained buying behaviors."
Sean DolanA Practical Look at Creating More Usable Enterprise Customer Journeys
October 31, 2019
"None of this matters without outcomes; we design to make people’s days better, not just to design."
Adam CutlerPeople + Places + Practices = Outcomes
June 8, 2016
"I don't want a component library for the account home. I want a system for all of that."
Nathan CurtisBeyond the Toolkit: Spreading a System Across People & Products
June 9, 2016
"Don’t be ideal. Do better. Start by making a public goal to create accountability and internal conversations."
Saara Kamppari-MillerInclusive Design is DesignOps
September 29, 2021
"Military contractors didn’t have design teams; most had never invested in UX design before."
Michal Anne RogondinoSaving Outer Space: The First UX Design System for Our Nation’s Satellites
January 8, 2024
"Leading with curiosity, tell me more, why is that? helps us see how research can be repositioned."
Feyikemi AkinwolemiwaUninvited, Not Unwelcome: What Researchers Bring When Nobody Asked
August 19, 2026
"What is your theory of change? How are you actually going to achieve the outcome you’re aiming for?"
Max Gadney Andrea Petrucci Joshua Stehr Hannah WickesAssessing UX jobs for impact in climate
August 14, 2024
