Summary
Jeff recounts the Ask Alexis project, a promising advice service launched by a team in New York that ultimately failed due to lack of full-time commitment, illustrating the challenge of scaling part-time initiatives. Melissa shares a similar experience with a simplified payroll app at Intuit that was canceled after strategic concerns about disrupting existing products led to a loss of experimentation discipline. They discuss Netflix's usability experiment where a simple design unexpectedly outperformed expert-picked options, highlighting that users are often less proactive than assumed. The panel emphasizes the importance of cultural acceptance of failure and humility even among experts, and the need to hire team members who thrive on business constraints and hypothesis-driven work. Bill and Alyssa add insights on prototyping strategy and overcoming internal company barriers to experimentation, including tactics for restricted corporate environments and sustaining behavioral change. The speakers converge on the necessity of strong vision paired with openness to data, continuous iteration, and understanding stakeholder perspectives to foster successful innovation.
Key Insights
-
•
Partial team commitment can doom even promising projects like Ask Alexis, underscoring the need for dedicated resources to scale.
-
•
Experimentation success can be overturned when strategic business pressures refocus teams away from data-driven iteration, as seen in Melissa's payroll app case.
-
•
Users often prefer simpler, less customizable experiences, contrary to expert expectations, as demonstrated by Netflix's simple grid winning over more complex UX.
-
•
Organizational culture must accept being wrong openly to enable iterative product success and honest experimentation.
-
•
Hiring emphasizes designers and product people who thrive under constraints and can think in hypotheses rather than simply artistic expression.
-
•
Prototyping should be aligned with what the team needs to learn next, whether that is validating value or testing technical performance.
-
•
Throwaway and evolutionary prototypes both have roles; balancing speed of ideation and closeness to production is critical.
-
•
Sustained behavioral change experiments require framework and measurement designed for longer-term user engagement rather than immediate clicks.
-
•
In restrictive enterprise environments, small internal experiments and ally-building are key to expanding a culture of testing.
-
•
Effective evangelism of experimentation depends on tailoring communication to the audience, whether executives, engineers, or designers.
Notable Quotes
"We had to kill the Ask Alexis product because the level of commitment needed couldn't happen with part-time consultants."
"After the payroll app was on the roadmap, folks started asking where's the revenue, and then they wanted to change the direction without experimentation."
"The Netflix grid experience that was simplest and offered no genre picking actually won, showing users are lazier than they think."
"Just because you've been right in the past doesn't guarantee you'll be right in the future."
"We hired designers who speak product, think in hypotheses, and love constraints rather than just artistic ideas."
"You have to ask yourself, what's the least amount of work you need to do to get the learning you want from a prototype."
"The developer who refused to run experiments bragged about success, but it was actually a flop once we ran the test properly."
"If you're changing workflows in experiments, you need to keep track of impacted teams like call centers to avoid resistance."
"Finding allies within an enterprise and demonstrating success is how you get a foothold for experimentation culture."
"Experimentation pitching must use the language and values of who you're trying to convince in the organization."
Or choose a question:
More Videos
"Consistency is so important that sometimes even consistency in failure works if it means I only have to learn the workaround once."
Sam ProulxOnline Shopping: Designing an Accessible Experience
June 7, 2023
"Career managers develop close relationships with designers and advocate for staffing and promotions on their behalf."
Ignacio MartinezFair and Effective Designer Evaluation
September 25, 2024
"Transformation is metamorphosis; it's often messy and aims to create systems out of chaos."
Sarah Kinkade Mariana Ortiz-ReyesDesign Management Models in the Face of Transformation
June 8, 2022
"The Shaker village sidewalks followed people’s natural inclinations, not forcing unnatural movement."
Daniel GloydWarming the User Experience: Lessons from America's first and most radical human-centered designers
May 9, 2024
"AI systems can present speculative connections as established facts, so confidence ratings are critical."
Patrick BoehlerFishing for Real Needs: Reimagining Journalism Needs with AI
June 10, 2025
"There’s a big merger in this space and two UX giants haven’t been behaving nicely, introducing higher prices and stricter commercial terms."
Andy Barraclough Betsy NelsonFrom Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
September 23, 2024
"I’m totally freaked out by facial recognition and wonder why we don’t get together as consumers to push back on bad designs."
Alexandra SchmidtWhy Ethics Can't Save Tech
November 18, 2022
"A lot of DevOps is about empathy — doing our piece but thinking about the larger connection."
Louis RosenfeldDiscussion: What Operations can teach DesignOps
November 6, 2017
"Carbon is just the starting point for what can be imagined. We're hired for our expertise to adapt, not just follow directions."
Mitchell BernsteinOrganizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
September 29, 2021
Latest Books All books
Dig deeper with the Rosenbot
Why is competitiveness declining as a motivation in gaming and product design, according to Cheryl Platz?
How does the tech industry's performance calibration cycle influence career longevity and mindset?
In what ways can organizations use scaffolding to overcome bureaucratic constraints in service design?