Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Get Your Whole Team Testing to Design for Impact
Wednesday, July 24, 2024 • Rosenfeld Community
Share the love for this talk
Get Your Whole Team Testing to Design for Impact
Speakers: Erin Weigel
Link:

Summary

Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.

Key Insights

  • Conversion design is a combination of user-centered design, scientific experimentation, and business strategy.

  • Historical context shows poor experimental design led to unreliable data and necessitated the development of robust methodologies.

  • The conversion design process is cyclical, allowing teams to iterate between understanding, hypothesizing, prioritizing, creating, testing, analyzing, and deciding.

  • Ethical considerations are crucial in design; experiments should not exploit or mislead users.

  • A/B testing is a fundamental part of the conversion design process, providing a reliable evidence base for decision-making.

  • Randomization is key to isolating the effects of changes and ensuring data validity in experiments.

  • Conversion design aims to create intentional improvements, not just differences, in user experience.

  • Measurement of success should include both quantitative metrics, like sales, and qualitative impacts, like user satisfaction.

  • An experimentation culture fosters growth and better decision-making across teams.

  • Documentation of experimental processes is vital for understanding implementation success and learning from outcomes.

Notable Quotes

"Design is the rendering of intent."

"Conversion means change, not just sales."

"When you do things, you should also think about the ethics of your decisions."

"You cannot purely test an idea, you can only ever test the execution of that idea."

"The smallest poor design choice can make an entire experiment fail."

"Data behaves like a metric whack-a-mole, and without good design, bias can influence results."

"You need to capture not just customer needs but the business goals tied to them."

"Experimentation is the main way to create high quality decisions."

"The conversion design cycle is the engine of how businesses function."

"Knowledge is the main output of the conversion design process."

More Videos

Amy Jiménez Márquez

"Jackie Ho emphasized the importance of aligning design teams while maintaining accountability."

Amy Jiménez Márquez Michael J. Metts Joie Chung

The Atypical UX Manager Path

July 23, 2020

Dave Gray

"Break a pattern, and you'll create new possibilities."

Dave Gray

Liminal Thinking: Sense-making for systems in large organizations

May 14, 2015

Sarah Barrett

"AI isn't the answer to all problems; it's about finding the right context for its use."

Sarah Barrett

AI in Real Life: Using LLMs to Turbocharge Microsoft Learn

February 13, 2025

Sam Ladner

"Set the participant at ease, and don't add to their burden."

Sam Ladner Megan Blocker Tanvi Dhond Katie Hansen Ricardo Martins

Methodologies: Beyond the interview [Advancing Research Community Workshop Series]

September 4, 2024

Lija Hogan

"The research we do needs to include how various bots can help each other."

Lija Hogan Milan Mijatovic Sam Proulx Louis Rosenfeld

Three Years Out: Perspectives on the Near-Term Future of User Research

March 15, 2024

"Our design ops team is about stretching the boundaries of what the role can do."

Panel Discussion: Communicating the Value of DesignOps

November 7, 2018

Sarah Rink

"Engagement from stakeholders can lead to unforeseen disruptions during research sessions."

Sarah Rink

Remote User Research: Dos and Don'ts from the Virtual Field

June 11, 2020

Melissa Tsang

"Values can be perceived as just fancy words on a wall, but they must be operationalized."

Melissa Tsang

From Insights to Action: Driving Business Values through DesignOps

January 8, 2024

Taylor Jennings

"Having all our data centralized helps facilitate quick insights."

Taylor Jennings Joe Nelson Alex Knoll

Repository Retrospective: Learnings from Introducing a Central Place for UX Research

March 9, 2022