Summary
The secret ingredient for impactful AI products is “evals”—an architecture for ongoing evaluation of quality. Without evals, you don’t know if your output is good. You don’t know when you’re done. Because outputs are non-deterministic, it’s very hard to figure out if you are creating real value for your users, and when something goes wrong, it’s really tricky to figure out why. Simply Put’s Peter van Dijck will demystify evals, and share a simple framework for planning for and building useful evals, from qualitative user research to automated evals using LLMs as a judge.
Upcoming events
Building impactful AI products for design and product leaders, Part 1: The new product journey
Building impactful AI products for design and product leaders, Part 2: Evals are your moat
More Videos
"AI is perceived as the shiny, sparkly thing while many undervalue the practical applications that truly address user needs."
Dan SafferWhy AI projects fail (and what we can do about it)
May 14, 2025
"The maps are just a model of the world; models are simple versions of the world."
Victor Lombardi Ted Booth HK Dunston Andrew OtwellBridging Design and Climate Science
February 14, 2024
"A research repository is not simply a storage space; it requires ongoing engagement."
Ben Davies Matt Duignan Andrew Michael Dr. Emily DiLeoExpert Panel: The Principles of Research Repository Design
March 11, 2022
"Dan was my first boss hired me out of Carnegie Mellon to come work for him at Oracle."
Daniel J. RosenbergDigital Medicine Design
September 26, 2019
"Leaders need to engender trust, empower others, and create the conditions for varied people to thrive."
Etienne FangThe Power of Care: From Human-Centered Research to Humanity-Centered Leadership
March 10, 2021
"To maximize our mission, we focused on aligning staffing to strategic priorities while being transparent throughout."
Patrick CommarfordDesign Staffing for Impact
January 8, 2024
"Values are the core assumption of what we are doing here."
Adam CutlerPeople + Places + Practices = Outcomes
June 8, 2016
"The key to accessibility is that it requires flexible designs, not limited designs."
Sam ProulxAccessibility: An Opportunity to Innovate
March 9, 2022
"The rate of change and capability of these technologies is unprecedented."
Husani OakleyTheme Three Intro
June 6, 2023
