Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

UX Lessons from running more than 1,200 A/B Tests
Wednesday, July 10, 2024 • Rosenfeld Community
Share the love for this talk
UX Lessons from running more than 1,200 A/B Tests
Speakers: Erin Weigel
Link:

Summary

Knowing how to solve the right problem is only one small part of the product-success equation. And nailing the execution is harder than most people think. As Erin says in this talk, “There are far more ways to fail than there are to succeed” when it comes to experimentation. To help you learn, Erin shares some of the silly mistakes she made while A/B testing her designs. That way you can avoid those pitfalls as you work to make your digital product better—not just different.

Key Insights

  • Nine out of ten AB tests at Booking.com fail, but the few that succeed compound into massive growth.

  • Execution often fails on good ideas rather than the ideas themselves.

  • Technical details like page load time can negate positive design changes if not carefully managed.

  • Edge case bugs in different languages or currencies can silently kill conversions.

  • Minor typography choices, such as switching from Times New Roman to Arial, have a large impact on conversion.

  • Using system fonts improves legibility and page load speed, positively affecting conversions.

  • Tracking should only include users exposed to the tested change to avoid noisy data.

  • Qualitative user research drives many experiment ideas by revealing real user challenges.

  • Guardrail metrics like customer service calls and loyalty prevent short-term wins from harming long-term value.

  • Products should be retested over time as technology and user behaviors evolve, since what fails today may succeed tomorrow.

Notable Quotes

"I’m actually a really big failure — nine out of ten tests fail with no positive measurable impact."

"Compound effect means good upon good upon good eventually builds to incredibly fast growth."

"If a problem keeps coming up, it’s usually the execution of the idea that’s failing, not the concept."

"Increasing page load time by three seconds can nullify all your design improvements."

"Picking the wrong size image can impact customer experience as much as what you see on screen."

"There are as many versions of the website as stars in the sky because of language, currency, and device variations."

"No tracking is perfect because numbers tell you what likely happened, but not why it happened."

"System fonts load faster, improve legibility, and outperform fancy brand fonts in conversion."

"Guardrail metrics help catch unintended consequences like increased customer service calls despite higher conversions."

"You can’t fail 90 times out of 10 without laughing at yourself to keep going."

Ask the Rosenbot
Andy Warr
Under My (Research) Umbrella: The Benefits and Challenges of Building a Unified Insights Function
2024 • Advancing Research 2024
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold
Louis Rosenfeld
Opening Remarks
2023 • Advancing Research 2023
Gold
Erika Kincaid
Connecting the Dots: How to Foster Collaboration and Build a Strong Design Review Culture
2022 • Design at Scale 2022
Gold
Joshua Graves
We Need To Talk: Managing Ludicrous Requests at Work (Part 3 of 3)
2025 • Rosenfeld Community
Frances Yllana
DesignOps–Leading the Path to Parity
2023 • DesignOps Community
Prayag Narula
How to Empower Your Designers to Do Good Research – And Why You Want To
2022 • Design at Scale 2022
Gold
Angelos Arnis
State of DesignOps: Learnings from the 2021 Global Report
2021 • DesignOps Summit 2021
Gold
Caroline Jarrett
Have fun with statistics?
2024 • Rosenfeld Community
Nathan Curtis
Beyond the Toolkit: Spreading a System Across People & Products
2016 • Enterprise UX 2016
Gold
JJ Kercher
A Roadmap for Maturing Design in the Enterprise
2018 • Enterprise Experience 2018
Gold
Corey Long
Hiring in DesignOps: A Critical Study on How to Hire and Get Hired
2024 • DesignOps Summit 2024
Gold
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Jackie Velasquez-Ross
Talent Acquisition and Our Responsibility
2020 • DesignOps Community
Dr. Jamika D. Burge
Broad Strokes: Connecting Design, Research, and AI to the World Around Us
2023 • Enterprise UX 2023
Gold

More Videos

Brian T. O’Neill

"Decision culture is a better lens than data culture for thinking about what decisions we're trying to facilitate with AI."

Brian T. O’Neill Maria Cipollone Luis Colin Manuel Dahm Mike Oren

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?

February 18, 2022

Jorge Arango

"I used GPT-4 to retag 1,200 blog posts, reducing an estimated 10–12 hours of tedious manual work to about a third of that time."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024

Bria Alexander

"If you ever feel unsafe or threatened, please let us know in the #do-help channel."

Bria Alexander

Opening Remarks

October 1, 2021

Louis Rosenfeld

"Our curators have been working for months taking presentations from little ideas to something big that advances design ops as practice."

Louis Rosenfeld Bria Alexander

Day 1 Welcome

September 23, 2024

Mansi Gupta

"Women are often treated as a minority, which sets them up to be forgettable, dispensable, and ignorable."

Mansi Gupta

Women-Centric Research: What, Why, How

March 29, 2023

Bria Alexander

"Sponsor sessions do not overlap at all with the main programming. It is 100% fully optional."

Bria Alexander

Opening Remarks

October 3, 2023

Andy Polaine

"Lots of service design has been happening but in disguise and not called that."

Andy Polaine Lavrans Løvlie

What is the role of service design in product-led organizations?

December 3, 2024

George Abraham

"This is a one-directional link between your design tool and the app builder; design tools don’t get updated by the builder."

George Abraham Stefan Ivanov

Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)

October 1, 2021

Robin Beers

"Power core is where senior executives really care about outcomes or functions that hold power."

Robin Beers

Beyond Insights: Researchers as Organizational Change Catalysts

March 25, 2024