Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Tess Dixon
C'mon Get Happy
2021 • DesignOps Summit 2021
Gold
Vasileios Xanthopoulos
A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale
2024 • Enterprise Experience 2020
Gold
Dominique Ward
The Most Exciting Time for DesignOps is Now
2022 • DesignOps Summit 2022
Gold
Josh Clark
Sentient Design: Crafting Intelligent Interfaces with AI
2026 • Rosenfeld Community
Zariah Cameron
ReDesigning Wellbeing for Equitable Care in the Workplace
2024 • DesignOps Summit 2024
Gold
Aditi Ruiz
Pulse Check: Empathy Mapping Your Product Manager, Pt. 2
2022 • Design in Product 2022
Gold
Kara Kane
Communities of Practice for Civic Design
2022 • Civic Design Community
Dan Saffer
Why AI projects fail (and what we can do about it)
2025 • Rosenfeld Community
Anupama Dhareshwar
From blueprint to bot: Designing resilient AI-powered services
2025 • Advancing Service Design 2025
Gold
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Lisa Spitz
Building Trust Through Equitable Research Practices
2022 • Civic Design 2022
Gold
Joy KendiMwiti
Deciding when to automate: Integrating AI in high-stakes systems
2026 • Designing with AI 2026
Conference
Louis Rosenfeld
Do you want to work on climate? (Climate UX Discussion Series)
2023 • Climate UX Interest Group
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Advancing Research 2022
Gold
Jason Mesut
Shaping design, designers and teams
2018 • DesignOps Summit 2018
Gold
Llewyn Paine
Day 1 Using AI in UX with Impact
2025 • Designing with AI 2025
Gold

More Videos

Greg Nudelman

"If a bot can’t handle the query, it goes to a human—it's like Mechanical Turk for AI."

Greg Nudelman

Designing Conversational Interfaces

November 14, 2019

Bud Caddell

"This is all wheat, no shafts, lessons drenched in hard work and real experience."

Bud Caddell

Theme 2 Intro

September 30, 2021

Billy Carlson

"ESPN’s site is a great example of poor hierarchy and alignment — it’s hard to know where to start or where to go next."

Billy Carlson

Pro-level UI Tips for Beginners

September 9, 2022

Robert Fabricant

"We need to reframe compensation around reciprocal exchange, not just monetary incentives."

Robert Fabricant Sahibzada Mayed Nidhi Singh Rathore

Industry junctures: Paths forwards for UXR and the critical decisions that get us there [Advancing Research Community Workshop Series]

October 2, 2024

Gillian Salerno-Rebic

"It’s surprisingly easy to bring in your own UX or marketing personas to run simulated testing with VO Pulse."

Gillian Salerno-Rebic Mark Micheli

From Insight to Impact: How JourneySpark Used WEVO Pulse + Pro to Drive a 50% Lift in Ad Engagement

June 11, 2025

Victor Lombardi

"Designers bring a unique cultural perspective to multidisciplinary scientific teams, helping interpret and communicate complex data."

Victor Lombardi Ted Booth HK Dunston Andrew Otwell

Bridging Design and Climate Science

February 14, 2024

Lada Gorlenko

"Leading change in a century-old company is a very different challenge."

Lada Gorlenko

Theme 1: Intro

January 8, 2024

Dan Willis

"The more specific a story gets, the more universal the themes are felt by everybody."

Dan Willis

Enterprise Storytelling Sessions

June 3, 2019

Emily Williams

"My race has been treated like guinea pigs for most of our existence."

Emily Williams Nora Fiore

When UX Research and Institutional Racism Collide: A Case Study

March 12, 2021