Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact
Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design combines design, science, and business to intentionally create measurable improvements, not just arbitrary changes.

  • Ronald Fisher’s 1919 work exposed how poor experimental design led to decades of unreliable scientific data.

  • Conversion originates from the Latin word meaning to transform or change, which is broader than just sales or profit.

  • Traditional linear product development processes miss the complex, iterative nature of real-world systems.

  • Systems thinking offers a more accurate way to understand and manage design experiments within interconnected environments.

  • A rigorous conversion design process includes seven phases: understand, hypothesize, prioritize, create, test, analyze, and decide.

  • Experiments grounded in well-documented hypotheses based on research have higher success rates than guesses.

  • Randomized 50/50 AB testing is the gold standard for isolating the true effect of design changes by evenly distributing confounds.

  • Experimentation buckets—product foundations, content/motivation, accessibility/usability, and bug fixes—help prioritize work effectively.

  • Ethical considerations and guardrail metrics are crucial to ensure changes benefit all stakeholders sustainably and without manipulation.

Notable Quotes

"Conversion means change, not just sales or profit."

"Design is the rendering of intent — bringing ideas into form that solve the problem."

"Decades worth of agricultural experimental data was garbage because of poor experimental design."

"Most product teams stay on the bottom rungs of evidence, relying on opinions or observational data instead of randomized trials."

"You can never purely test an idea, only the implementation of the idea."

"Randomization is magic — it evenly distributes confounds so the observed effect is caused by your change."

"Not all changes create value; some do nothing or even make things worse."

"You have to think critically about how a change impacts all stakeholders, not just the main business metric."

"Ethics evolve faster than laws; just because something is legal doesn’t make it ethical."

"If an experiment I design made the front page news tomorrow, how would I feel about it?"

Ask the Rosenbot
Harry Brignull
Beyond Clicks and Tricks: Why deceptive design has grown into a regulatory faultline
2026 • Rosenfeld Community
Benjamin Wiedmaier
Redefining Toolkits: Unbundling to Create a Perfect Match
2025 • Advancing Research 2025
Gold
Bria Alexander
Opening Remarks
2022 • DesignOps Summit 2022
Gold
Amanda Kaleta-Kott
The Joys and Dilemmas of Conducting UX Research with Older Adults
2022 • Advancing Research 2022
Gold
Louis Rosenfeld
Opening Remarks
2023 • DesignOps Summit 2023
Gold
Megan Clegg
Space for Everyone: Reframing Accessibility Through a Wider Lens
2021 • Design at Scale 2021
Gold
Yasmine Khan
Checking Bias and Listening to Financially Vulnerable Americans
2020 • Advancing Research 2020
Gold
Robin Beers
How to create actionable insight in the face of politics and silos [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Jorge Arango
The Best of Both Worlds: How to Integrate Paper and Digital Notes (1st of 3 seminars)
2024 • Rosenfeld Community
Barb Spanton
Doing Work That Matters: A Look Beyond The Idealistic Notion of 'Doing Meaningful Work'
2022 • Design at Scale 2022
Gold
Nathan Curtis
Discussion
2016 • Enterprise UX 2016
Gold
Malini Rao
Lessons Learned from a 4-year Product Re-platforming Journey
2021 • Design at Scale 2021
Gold
Abby Covert
Stuck? Diagrams Help
2022 • DesignOps Community
Megan Nipe
Human-Centered Design for Engagement: Maturing from Newsletterville to Personalized, One-to-One Messaging
2021 • Civic Design 2021
Gold
Dr Chloe Sharp
Using Evidence and Collaboration for Setting and Defending Priorities
2023 • Design in Product 2023
Gold
Tamara Kartoziia
Think global, adapt local: how service design accelerated B2B market entry by 6 months
2025 • Advancing Service Design 2025
Conference

More Videos

John Taschek

"I’d rather be known for bringing humanity into the tech world than anything else I’ve done."

John Taschek

Making People the X-Factor in the Enterprise

June 15, 2018

Jessamyn Edwards

"Tracking complaints and fixing things immediately, even if it’s just one person, is a mark of good UX impact."

Jessamyn Edwards

Surviving Your UX Career in Enterprise Design

December 2, 2021

Karen McGrane

"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."

Karen McGrane Jeff Eaton

AI for Information Architects: Are the robots coming for our jobs?

November 21, 2024

Wendy Johansson

"We don’t call anyone a product manager in our org, but as a product boss, I wear that hat and help designers grow into that ownership."

Wendy Johansson

Be a Product Boss!

December 6, 2022

Mike Oren

"Research repositories often get a bad rap because people just dump final reports without thinking about how they'll be used."

Mike Oren Janice Wiitala

Design Research Strategy & Strategic Design Research

February 3, 2022

Patrick Boehler

"Humans remain essential for evaluation, especially to interpret cultural nuance that AI often misses."

Patrick Boehler

Fishing for Real Needs: Reimagining Journalism Needs with AI

June 10, 2025

Dave Gray

"The higher up you are in an organization, the more distorted your reality is likely to be."

Dave Gray

Liminal Thinking: Sense-making for systems in large organizations

May 14, 2015

Scott Stephens

"Coda doesn’t charge for editors, only for document makers, making it much more affordable to scale in large organizations."

Scott Stephens

The Next Generation in DesignOps Toolsets

July 28, 2022

Michael Polivka

"Design is frequently misunderstood; sometimes it’s just about explaining why it’s important to get others involved."

Michael Polivka

Scaling Design through Relationship Maps

November 7, 2017