Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Kristen Honey
"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design
2021 • Civic Design Community
Josh Clark
Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
2025 • Rosenfeld Community
Samuel Proulx
From Standards to Innovation: Why Inclusive Design Wins
2025 • DesignOps Summit 2025
Gold
Francesca Barrientos, PhD
You Need Your Own Definition of Design Maturity
2022 • Design at Scale 2022
Gold
Kit Unger
Theme 2: Discussion
2024 • Enterprise Experience 2020
Gold
Sharon Bautista
Time to Make the Donuts: How User Research Helped Bridge Disparate Teams
2024 • Enterprise Experience 2020
Gold
Jemma Ahmed
Theme 2 Intro
2024 • Enterprise Experience 2020
Gold
Isaac Heyveld
Expand DesignOps Leadership as a Chief of Staff
2022 • DesignOps Summit 2022
Gold
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Gold
Rachael Dietkus, LCSW
Leading through the long tail of trauma
2022 • Enterprise Community
Jen Crim
Culture, DIBS & Recruiting
2021 • Design at Scale 2021
Gold
Susan Simon-Daniels
War Stories LIVE! Susan Simon-Daniels
2020 • Advancing Research 2020
Gold
Alnie Figueroa
Teamwork: Strategies for Effective Collaboration with Other Program Management Teams
2022 • DesignOps Summit 2022
Gold
Dawn Ressel
Full-Stack User Experiences: A Marriage of Design and Technology
2016 • Enterprise UX 2016
Gold
Nicole Aleong
Future Orientations to Everyday Life: Futures Anthropology as a Methodology
2024 • Advancing Research 2024
Gold

More Videos

Jon White

"Don’t try to tag all historical work at once — tag as you go to avoid re-platforming nightmares."

Jon White Erin May

Unsticking Research for Better Information Flow

March 11, 2026

Luz Bratcher

"There’s no map or measure of what is good enough in hustle culture."

Luz Bratcher

This Is a Talk for Tired People

June 10, 2022

Ignacio Martinez

"We want to build consultants who can solve problems across a variety of projects and clients."

Ignacio Martinez

Fair and Effective Designer Evaluation

September 25, 2024

Bria Alexander

"The pressure in design ops is often delivery-first mindset instead of quality and strategic impact - Dave Malu."

Bria Alexander Patrizia Bertini Peter Boersma Jon Fukuda Dave Malouf Theresa Slate Changying (Z) Zheng

Charting the future of DesignOps: A community workshop

April 18, 2024

Michael Weir

"If a website is hard to use, motivation and approach drops, and users are less likely to engage or convert."

Michael Weir

Mixed Methods and Behavioural Science

May 26, 2023

Sam Proulx

"Confidence is critical in online shopping because nobody wants to spend money they’re unsure about losing due to unclear controls."

Sam Proulx

Online Shopping: Designing an Accessible Experience

October 3, 2023

Alla Weinberg

"Stress drops our operating IQ by half, so we can’t do our jobs well under chronic stress."

Alla Weinberg

People Are Sick of Change: Psychological Safety is the Cure

July 20, 2023

Amy Paris

"If you don’t collect data on somebody, they become invisible and nobody should be invisible."

Amy Paris Danielle Thierry

Delivering Equity: Government Services for All Ages, Languages, Sexual Orientations, and Gender Identities

December 9, 2021

Francesca Barrientos, PhD

"Use your powers of empathy to see things from your functional partners’ point of view."

Francesca Barrientos, PhD

You Need Your Own Definition of Design Maturity

June 8, 2022