Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact
Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design combines design, science, and business to intentionally create measurable improvements, not just arbitrary changes.

  • Ronald Fisher’s 1919 work exposed how poor experimental design led to decades of unreliable scientific data.

  • Conversion originates from the Latin word meaning to transform or change, which is broader than just sales or profit.

  • Traditional linear product development processes miss the complex, iterative nature of real-world systems.

  • Systems thinking offers a more accurate way to understand and manage design experiments within interconnected environments.

  • A rigorous conversion design process includes seven phases: understand, hypothesize, prioritize, create, test, analyze, and decide.

  • Experiments grounded in well-documented hypotheses based on research have higher success rates than guesses.

  • Randomized 50/50 AB testing is the gold standard for isolating the true effect of design changes by evenly distributing confounds.

  • Experimentation buckets—product foundations, content/motivation, accessibility/usability, and bug fixes—help prioritize work effectively.

  • Ethical considerations and guardrail metrics are crucial to ensure changes benefit all stakeholders sustainably and without manipulation.

Notable Quotes

"Conversion means change, not just sales or profit."

"Design is the rendering of intent — bringing ideas into form that solve the problem."

"Decades worth of agricultural experimental data was garbage because of poor experimental design."

"Most product teams stay on the bottom rungs of evidence, relying on opinions or observational data instead of randomized trials."

"You can never purely test an idea, only the implementation of the idea."

"Randomization is magic — it evenly distributes confounds so the observed effect is caused by your change."

"Not all changes create value; some do nothing or even make things worse."

"You have to think critically about how a change impacts all stakeholders, not just the main business metric."

"Ethics evolve faster than laws; just because something is legal doesn’t make it ethical."

"If an experiment I design made the front page news tomorrow, how would I feel about it?"

Ask the Rosenbot
Jack Moffett
UX Metrics That Matter and The Future of our Design at Scale Conference: A Community Conversation
2022 • Enterprise Community
Kristin Wisnewski
Measuring What Matters
2019 • DesignOps Summit 2019
Gold
Taiye Akin-Akinyosoye
Amplifying voices and enhancing user research through group interviews
2025 • Advancing Research 2025
Gold
Llewyn Paine
Coexisting with AI: A practical guide for researchers to navigate tools, ethics, and integration
2025 • Advancing Research 2025
Gold
Megan Clegg
Space for Everyone: Reframing Accessibility Through a Wider Lens
2021 • Design at Scale 2021
Gold
Sam Proulx
SUS: A System Unusable for Twenty Percent of the Population
2021 • Civic Design 2021
Gold
Cheryl Platz
Embrace Your Fun Factor: Game Development Best Practices for Product Design
2026 • Rosenfeld Community
Jennifer Strickland
Adopting a "Design By" Method
2021 • Civic Design 2021
Gold
Shipra Kayan
Emerging principles for using AI in Design: What the product design team at Miro has learned from deeply integrating AI in their workflow
2025 • Designing with AI 2025
Gold
Andrea Gallagher
The Problem Space
2019 • Advancing Research Community
Bria Alexander
The Big Question about Resilience: A panel discussion
2024 • DesignOps Summit 2024
Gold
Erika Flowers
The Handoff is Dead: Design-Led Engineering with AI Agents
2026 • Rosenfeld Community
Erin Weigel
Real-world lessons to improve your conversion rates
2024 • Rosenfeld Community
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Rosenfeld Community
Patrizia Bertini
Designing Within the Lines: How the EU AI Act Can Spark Better AI Innovation
2025 • DesignOps Community
Laura Klein
Unique challenges of innovation in enterprises
2020 • Enterprise Community

More Videos

Alba Villamil

"If you don’t have a state ID, many social services become inaccessible, showing how resource barriers matter."

Alba Villamil

Stereotyped by Design: Pitfalls in Cross-Cultural User Research

March 30, 2020

Jake Burghardt

"Contributors to insight summaries should get more value out of it than they put into it."

Jake Burghardt

Stop wasting research: Create new value with insight summaries

July 9, 2025

Dr. Nikki Smith

"I can tell when something is real when multiple teams are talking about it and planning together."

Dr. Nikki Smith

Research Strategy: Connecting Insights to Outcomes

March 12, 2025

Erin Weigel

"Sometimes laziness can be a virtue if it leads to big impact with little work."

Erin Weigel

Real-world lessons to improve your conversion rates

June 26, 2024

Ana Maria Montero Barrantes

"For the right audience, if applied correctly, humor and music can move people in many ways."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Megan Blocker

"This moment is yours to shape. If you step up to it, you won’t just be expanding your toolkit. You’re going to be expanding your influence, your impact, and the future of our profession itself."

Megan Blocker

Theme 2 Intro

March 12, 2025

Bruce Gillespie

"Weave in great user stories that exemplify your key points to maximize engagement."

Bruce Gillespie

Learning from journalism: Balancing impactful communication with compassionate storytelling

March 13, 2025

Nathan Shedroff

"Business people thrive in certainty; designers habitually thrive in ambiguity."

Nathan Shedroff

Redefining Value: Bridging the Innovation Culture Divide

May 14, 2015

JJ Kercher

"Our team mission is to accelerate customer value through exceptional end-to-end experiences."

JJ Kercher

A Roadmap for Maturing Design in the Enterprise

June 15, 2018