Summary
The secret ingredient for impactful AI products is “evals”—an architecture for ongoing evaluation of quality. Without evals, you don’t know if your output is good. You don’t know when you’re done. Because outputs are non-deterministic, it’s very hard to figure out if you are creating real value for your users, and when something goes wrong, it’s really tricky to figure out why. Simply Put’s Peter van Dijck will demystify evals, and share a simple framework for planning for and building useful evals, from qualitative user research to automated evals using LLMs as a judge.
Upcoming events
Building impactful AI products for design and product leaders, Part 1: The new product journey
Building impactful AI products for design and product leaders, Part 2: Evals are your moat
More Videos
"Designers play a key role in shaping the developer experience to enhance productivity in government agencies."
Ed MullenDesigning the Unseen: Enabling Institutions to Build Public Trust
November 16, 2022
"Organizations want to solve big existential, durable questions."
Peter LevinSolve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility
March 25, 2024
"We need to see OKRs as a journey, not just a concrete goal."
Bria Alexander Benson Low Natalya Pemberton Stephanie GoldthorpeOKRs—Helpful or Harmful?
January 20, 2022
"One of the coolest things an operations team can have is building the tools that we need to enable growth at scale"
Kate Koch Prateek KalliFlex Your Super Powers: When a Design Ops Team Scales to Power CX
September 30, 2021
"Our lenders want the best experience for our borrowers."
Amy MarquezINVEST: Discussion
June 15, 2018
"What you build matters, but what matters more is if you learn."
Brenna FallonLearning Over Outcomes
October 24, 2019
"Resilience should be supported by organizational structures, not just individuals."
Brian MossWhat Does it Mean to be a Resilient Research Team?
March 9, 2022
"We hold the mirror up to our product teams and also to every single one of our 25,000 employees."
Adel Du ToitGet Your CFO To Say: 'Our Strategic Goal is User Obsession'
June 10, 2022
"We love to flex our superpowers, but we’re trying to use them for good and not evil."
Kate Koch Jean-Claire FitschenFlex Your Super Powers: When a Design Ops Team Scales to Power CX
September 29, 2021
