Rosenverse
Get Your Whole Team Testing to Design for Impact

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact

Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design combines design, science, and business to intentionally create measurable improvements, not just arbitrary changes.

  • Ronald Fisher’s 1919 work exposed how poor experimental design led to decades of unreliable scientific data.

  • Conversion originates from the Latin word meaning to transform or change, which is broader than just sales or profit.

  • Traditional linear product development processes miss the complex, iterative nature of real-world systems.

  • Systems thinking offers a more accurate way to understand and manage design experiments within interconnected environments.

  • A rigorous conversion design process includes seven phases: understand, hypothesize, prioritize, create, test, analyze, and decide.

  • Experiments grounded in well-documented hypotheses based on research have higher success rates than guesses.

  • Randomized 50/50 AB testing is the gold standard for isolating the true effect of design changes by evenly distributing confounds.

  • Experimentation buckets—product foundations, content/motivation, accessibility/usability, and bug fixes—help prioritize work effectively.

  • Ethical considerations and guardrail metrics are crucial to ensure changes benefit all stakeholders sustainably and without manipulation.

Notable Quotes

"Conversion means change, not just sales or profit."

"Design is the rendering of intent — bringing ideas into form that solve the problem."

"Decades worth of agricultural experimental data was garbage because of poor experimental design."

"Most product teams stay on the bottom rungs of evidence, relying on opinions or observational data instead of randomized trials."

"You can never purely test an idea, only the implementation of the idea."

"Randomization is magic — it evenly distributes confounds so the observed effect is caused by your change."

"Not all changes create value; some do nothing or even make things worse."

"You have to think critically about how a change impacts all stakeholders, not just the main business metric."

"Ethics evolve faster than laws; just because something is legal doesn’t make it ethical."

"If an experiment I design made the front page news tomorrow, how would I feel about it?"

Ask the Rosenbot
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Aaron Stienstra
Leveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
2021 • Civic Design 2021
Gold
Richard Buchanan
Creativity and Principles in the Flourishing Enterprise
2018 • Enterprise Experience 2018
Gold
Jake Burghardt
Finding More Inroads into Research Impact
2026 • Rosenfeld Community
Amahra Spence
Designing for Liberation, Rehearsing Freedom
2022 • Civic Design 2022
Gold
Sam Proulx
Designing For Screen Readers: Understanding the Mental Models and Techniques of Real Users
2021 • Civic Design 2021
Gold
Maria Rosala
AI-Assisted Qualitative Analysis: What to Automate and What to Own
2026 • Rosenfeld Community
Mujtaba Hameed
Frameworks for Excellence: Using Visual Thinking and Communication to Elevate Your Research
2024 • Advancing Research 2024
Gold
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Shahrzad Samadzadeh
What Is My Value? Two Takes and Some Mistakes
2024 • Enterprise Experience 2020
Gold
Dantley Davis
Leadership & Diversity—A Fireside Chat with Dantley Davis
2020 • Enterprise Community
Alexis Lucio
Scaling Accessibility Through Design Systems
2022 • Design at Scale 2022
Gold
Sam Proulx
Everything You Ever Wanted to Know About Screen Readers
2021 • Design at Scale 2021
Gold
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Gold
Zariah Cameron
Streamlining an Inclusive Design Practice
2023 • DesignOps Summit 2023
Gold
Andy Barraclough
From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
2024 • DesignOps Summit 2024
Gold

More Videos

Victor Udoewa

"Whiteness isn’t about color; people of all colors can reinforce white supremacy."

Victor Udoewa

Beyond Methods and Diversity: The Roots of Inclusion

March 26, 2024

Kim Lenox

"I don’t want people to do a 6 AM call — I can do a 10 PM call to brainstorm but not a design critique."

Kim Lenox

Leading Distributed Global Teams

May 20, 2019

Jennifer Fraser

"The bridge was made between math and physics in a way that hadn’t been done before."

Jennifer Fraser

What would Emmy Noether Do? Math, Models and Mulling in UX Research

March 29, 2023

Gina Mendolia

"Mirroring content by repeating back what people say helps them understand their own ideas better."

Gina Mendolia

Therapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems

December 3, 2024

Bryce Benton

"You can copy and paste error messages into ChatGPT, and it can help you fix your scripts quickly."

Bryce Benton

[Demo] AI-powered UX enhancement: Aligning GitHub documentation with USWDS at Austin Public Library

June 4, 2024

James Lang

"That productivity revolution—the idea we don’t need more headcount but better tools—is more aspiration than reality at the moment."

James Lang

Hopeful Futures for UX Research

January 30, 2026

Jorge Arango

"I used GPT-4 to retag 1,200 blog posts, reducing an estimated 10–12 hours of tedious manual work to about a third of that time."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024

Erin May

"Two researchers can’t come close to digging into the customer problems for 17 product teams."

Erin May Roberta Dombrowski Laura Oxenfeld Brooke Hinton

Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org

March 10, 2022

Natalie Dunbar

"Establish success measures that show how content strategy contributes to design ops beyond individual projects."

Natalie Dunbar

DesignOps and Content Strategy: Envisioning the Future Together

October 1, 2021