Summary
If you’re a product manager, UX researcher, or any kind of designer involved in creating an AI product or feature, you need to understand evals. And a great way to learn is with a hands-on example. In this second talk in the series, Peter Van Dijck of the helpful intelligence company will show you how to create an eval for an AI product using an LLM as a judge (when we use a Large Language Model to evaluate the output of another Large Language Model). We’ll have a look at how that works, but also dig into why this even works. Are we creating problems for ourselves when we let an LLM judge itself? This talk is hands on; and there will be plenty of time for questions. You will go away understanding when and how to use LLM as a judge, and build some product sense around how the best AI products today are built, and how that can help you use them more effectively yourself.
Upcoming events
More Videos
"Dedicated staffing gives designers deep vertical mastery but limits horizontal experience."
Alicia MootyDesign Staffing Models
September 30, 2021
"Either you need accessibility now or you will need accessibility in the future."
Sheri Byrne-HaberAccessibility at Scale
June 9, 2021
"People confuse statistical significance with significance in practice — they want to know if the change is meaningful, not just mathematically significant."
Caroline Jarrett Erin WeigelHave fun with statistics?
December 12, 2024
"If you have questions, I will take questions throughout."
Kate KalcevichDesigning inclusively with AI
June 5, 2024
"Forgiveness is not about forgetting or just getting over something, it’s about making a mission out of it."
Jim KalbachPeace is waged with sticky notes: Mapping Real-World Experiences
June 14, 2018
"We’re moving from theory of change to theory of service: starting with what people actually need before creating anything."
Patrick BoehlerFishing for Real Needs: Reimagining Journalism Needs with AI
June 10, 2025
"Firsthand engagement with UX research shows that it doesn’t slow you down — it can actually help things move faster."
Hana NagelTurning Research Ripples into Waves
November 8, 2018
"We realized designing for AI is very different from other computational mediums we used before."
Aras BilgenWho does the math: A designer’s journey in building an AI-based tutoring app
June 10, 2025
"Investing in teaching from the ground up helps grow the design ops discipline for the future."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022
Latest Books All books
Dig deeper with the Rosenbot
How can you embed accessibility into product development workflows effectively?
Why is it important to include people with disabilities early in discovery and usability research?
How can European digital regulations like GDPR and the Accessibility Act impact US companies with products used in Europe?