Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Experimentation can be intimidating to non-data science folk. But Erin wants to get everyone excited about A/B testing. In this talk, Erin shares the Conversion Design process. It centers A/B testing as a way to gather high-quality evidence to make highly informed decisions to improve your digital product. She also introduces the Good Experimental Design toolkit. These easy-to-follow templates usher teams through the logic needed to design trustworthy experiments that you can learn from.
Key Insights
-
•
Conversion design is a combination of user-centered design, scientific experimentation, and business strategy.
-
•
Historical context shows poor experimental design led to unreliable data and necessitated the development of robust methodologies.
-
•
The conversion design process is cyclical, allowing teams to iterate between understanding, hypothesizing, prioritizing, creating, testing, analyzing, and deciding.
-
•
Ethical considerations are crucial in design; experiments should not exploit or mislead users.
-
•
A/B testing is a fundamental part of the conversion design process, providing a reliable evidence base for decision-making.
-
•
Randomization is key to isolating the effects of changes and ensuring data validity in experiments.
-
•
Conversion design aims to create intentional improvements, not just differences, in user experience.
-
•
Measurement of success should include both quantitative metrics, like sales, and qualitative impacts, like user satisfaction.
-
•
An experimentation culture fosters growth and better decision-making across teams.
-
•
Documentation of experimental processes is vital for understanding implementation success and learning from outcomes.
Notable Quotes
"Design is the rendering of intent."
"Conversion means change, not just sales."
"When you do things, you should also think about the ethics of your decisions."
"You cannot purely test an idea, you can only ever test the execution of that idea."
"The smallest poor design choice can make an entire experiment fail."
"Data behaves like a metric whack-a-mole, and without good design, bias can influence results."
"You need to capture not just customer needs but the business goals tied to them."
"Experimentation is the main way to create high quality decisions."
"The conversion design cycle is the engine of how businesses function."
"Knowledge is the main output of the conversion design process."
More Videos
"Thank you to the design ops Summit curation and conference team for making it all possible."
Jon FukudaTheme One Intro
September 8, 2022
"The notion that organizations suffer not from a lack of ideas but from cultural impediments is echoed by Bud."
Bud Caddell Kristin Skinner Alana WashingtonDesignOps Community Sensing Session
May 13, 2021
"Trust is the true bedrock of all healthy and happy cultures."
Laura WeissTurn Down the Heat: 3 Ways to Handle Conflict in the Moment
November 20, 2024
"Look for quick wins that create value for both PMs and the research team."
Smitha Papolu Nova Wehman-Brown Melissa Schmidt Adam MenterTheme 3 Discussion
June 4, 2019
"We need to consider how automation helps and hinders design work."
Clara Kliman-SilverUX Futures: The Role of Artificial Intelligence in Design
June 7, 2023
"Each of those sides contributes revenue to the bottom line in some way."
Amy MarquezINVEST: Discussion
June 15, 2018
"The way you tell the story of your research is critical to gaining traction."
Megan Blocker Mujtaba Hameed Victor UdoewaPanel: Excellence in Impact
March 25, 2024
"You create an environment where people feel okay to speak up and disagree, leading to healthy exchanges."
Steve Baty Richard Dalton Maria Giudice Harry MaxDiscussion
June 9, 2016
"Our mission is to facilitate and strengthen cross-disciplinary collaboration."
Amelia ColeData-Prompted Interviews
December 17, 2021
