Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests

Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Sarah Auslander
Incremental Steps to Drive Radical Innovation in Policy Design
2022 • Civic Design 2022
Gold
Benjamin Real
Showing the Value of DesignOps by Not Having a DesignOps Team
2020 • DesignOps Summit 2020
Gold
George Aye
Designing a New Social Contract
2026 • Rosenfeld Community
Kaitlin Tasker
Fast and Fearless Inclusive Research
2023 • Advancing Research 2023
Gold
Alla Weinberg
Design Teams Need Psychological Safety: Here’s How to Create It
2022 • DesignOps Summit 2022
Gold
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Michael Weir
Mixed Methods and Behavioural Science
2023 • Rosenfeld Community
Chris Hodowanec
Agile + User Experience: How to navigate the Agile landscape as an UX Practitioner
2022 • Civic Design 2022
Gold
Taylor Jennings
Repository Retrospective: Learnings from Introducing a Central Place for UX Research
2022 • Advancing Research 2022
Gold
Deanna Washington
Connecting the Ops: Plenary Panel and Closing Circle
2022 • DesignOps Summit 2022
Gold
Karen Pascoe
Developing Experience Teams and Talent in the Enterprise
2016 • Enterprise UX 2016
Gold
Alexis Lucio
Scaling Accessibility Through Design Systems
2022 • Design at Scale 2022
Gold
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • DesignOps Summit 2022
Gold
Steve Baty
Breaking Out of Ruts: Tips for Overcoming the Fear of Change
2016 • Enterprise UX 2016
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • Designing with AI 2025
Gold
Jemma Ahmed
Redefining the research toolkit: Expanding methodologies for a changing world
2025 • Advancing Research 2025
Gold

More Videos

Kritika Yadav

"The AI assistant starts with reasoning and not results, engaging users in intelligent conversation to understand needs."

Kritika Yadav

Optimizing AI Conversations: A Case Study on Personalized Shopping Assistance Frameworks

June 10, 2025

Peter Van Dijck

"Evals are everywhere, right? Everybody's talking about evals. It is like one of the key things in developing useful AI products."

Peter Van Dijck

Hands-on AI #2: Understanding evals: LLM as a Judge

October 15, 2025

Toby Haug

"Dual track agile means you have the dev sprint track and a separate design/product track that leads the dev track by several cycles."

Toby Haug

Discussion

June 9, 2017

Saara Kamppari-Miller

"We use storytelling to turn accessibility from just compliance into a business differentiator that executives care about."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Dave Malouf

"Do we have the feedback loops to feel our way through paths of emergence rather than pre-carved routes?"

Dave Malouf

Theme 3: Introduction and Provocation

January 8, 2024

Jodi Forlizzi

"Most designers join AI projects after the data has been discovered and the model built—that’s far too late for meaningful innovation."

Jodi Forlizzi

Design and AI innovation

June 5, 2024

Wyatt Hayman

"We decided to limit panel access initially to the research team but are exploring expanding to other teams."

Wyatt Hayman

Global Research Panels

August 8, 2020

Robin Beers

"I don’t care about what’s actionable for a company. I care about what’s actionable for my client’s career and influence."

Robin Beers Sonja Bobrowska Mujtaba Hameed Josh Morales

How to create actionable insight in the face of politics and silos [Advancing Research Community Workshop Series]

October 12, 2023

Bria Alexander

"Working on a different word than empower: People already have power, we have to respect it."

Bria Alexander Ariel Kennan Charlotte Lee Sarah Brooks Emily Lessard Gordon Ross Joanne Dong

Reflect and Chart Forward

December 10, 2021