Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests
Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ivana Ng
Level Up Your Program with ProductOps
2024 • Enterprise Experience 2020
Gold
Bill Scott
Lean Engineering: Engineering for Learning and Experimentation in the Enterprise
2015 • Enterprise UX 2015
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Design in Product 2023
Gold
Alla Weinberg
Cross-Functional Relationship Design
2022 • Design in Product 2022
Gold
Russ Unger
Onboarding: The Ecosystem, not the Afterthought
2017 • DesignOps Summit 2017
Gold
Sandra Camacho
Creating More Bias-Proof Designs
2025 • Rosenfeld Community
Bria Alexander
Opening Remarks
2023 • Advancing Research 2023
Gold
Roy Opata Olende
How Zapier Uses ‘All Hands Research’ to Increase Exposure to Users
2020 • Advancing Research Community
Victor Lombardi
Bridging Design and Climate Science
2024 • Climate UX Interest Group
Luz Bratcher
This Is a Talk for Tired People
2022 • Design at Scale 2022
Gold
Kyria Stephens
Power to Heal: Civic Design in the Aftermath of Tragedy
2022 • Civic Design 2022
Gold
Bas Raijmakers, PhD (RCA)
What Design Research can Learn from Documentary Filmmaking
2022 • Advancing Research 2022
Gold
Bria Alexander
Opening Remarks Day 2
2024 • Advancing Research 2024
Gold
Ashley Cortez
Shifting Toward Community-Led Innovation in Local Government
2021 • Civic Design 2021
Gold
Caroline Jarrett
Have fun with statistics?
2024 • Rosenfeld Community
Sha Hwang
The First Fifty Years of Civic Design
2022 • Civic Design 2022
Gold

More Videos

Yoel Sumitro

"Researchers were obsessed with scientific rigor while designers used balance, feelings, or intuition as their rigor."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Zariah Cameron

"Share everything, own nothing but credit everyone."

Zariah Cameron

ReDesigning Wellbeing for Equitable Care in the Workplace

September 23, 2024

Doug Powell

"Forrester found that design thinking teams move twice as fast from idea to market because they’re closer to their users."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Darian Davis

"Creating a roadmap built enough space to make meaningful design changes for developers to implement."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Mike Oren

"New ideas and changes to business trajectories rank pretty far down at only 18% of business leaders seeing it as a value of design."

Mike Oren

Why Pharmaceutical's Research Model Should Replace Design Thinking

March 28, 2023

Jacqui Frey

"Design Ops brings the joy back into design."

Jacqui Frey

Scale is Social Work

March 19, 2020

Kim Holt

"Showing gratitude through Slack workflows is a low effort, high impact way to build community and recognize everyday wins."

Kim Holt Emma Wylds Pearl Koppenhaver Maisee Xiong

A Salesforce Panel Discussion on Values-Driven DesignOps

September 8, 2022

Samuel Proulx

"When we design for the edges, we get the middle for free."

Samuel Proulx

From Standards to Innovation: Why Inclusive Design Wins

September 10, 2025

Laine Riley Prokay

"For early career UXers, compassion could be as simple as expressing gratitude for help."

Laine Riley Prokay

How DesignOps can Drive Inclusive Career Ladders for All

September 30, 2021