Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Hands-on AI #2: Understanding evals: LLM as a Judge

Wednesday, October 15, 2025 • Rosenfeld Community

This video is featured in the Evals + Claude playlist.

Share the love for this talk
Hands-on AI #2: Understanding evals: LLM as a Judge
Speakers: Peter Van Dijck
Link:

Summary

If you’re a product manager, UX researcher, or any kind of designer involved in creating an AI product or feature, you need to understand evals. And a great way to learn is with a hands-on example. In this second talk in the series, Peter Van Dijck of the helpful intelligence company will show you how to create an eval for an AI product using an LLM as a judge (when we use a Large Language Model to evaluate the output of another Large Language Model). We’ll have a look at how that works, but also dig into why this even works. Are we creating problems for ourselves when we let an LLM judge itself? This talk is hands on; and there will be plenty of time for questions. You will go away understanding when and how to use LLM as a judge, and build some product sense around how the best AI products today are built, and how that can help you use them more effectively yourself.

Key Insights

  • Evals are a foundational feedback loop defining what 'good' means for AI products, helping to measure and improve systems continuously.

  • Evaluating fuzzy, subjective AI outputs requires innovative approaches such as using LLMs as judges to score results.

  • Binary (yes/no) scoring is more reliable than rating scales with ranges because LLMs lack internal memory and consistency.

  • Starting evals early (week one of a project) drastically improves AI product outcomes, but many teams delay due to perceived complexity.

  • High-risk or important tasks should be prioritized for evals instead of attempting broad coverage.

  • Assigning a dedicated owner or 'benevolent dictator' for evals who works closely with domain experts accelerates feedback and quality.

  • Creating a written constitution of principles helps concretize AI behavior goals and guides prompt and model training.

  • Most current eval tooling is too technical, slowing iteration cycles and making expert involvement inefficient.

  • Custom feedback interfaces tailored to expert users significantly speed up evaluating AI outputs in domains like healthcare and law.

  • Diverse perspectives from UX, product, strategy, and domain experts are critical in defining and refining what 'good' means in AI systems.

Notable Quotes

"Evals are everywhere, right? Everybody's talking about evals. It is like one of the key things in developing useful AI products."

"You want to ask an LLM to evaluate the fuzzy stuff because there’s no black and white output."

"LLMs don’t have memory, so rating on a scale from one to five is pretty random. Better to have yes or no answers."

"One of the biggest problems in AI building is evolving your prompts and having a fast feedback loop."

"By starting to categorize risk in detail, you naturally lead to better prompts and better evals."

"A constitution is a very good exercise: write down your system’s principles and values to help guide its behavior."

"Use custom systems for experts to quickly review and rate outputs, making feedback cycles much faster."

"Evals define a shared definition of good with tests to measure it, and that is the secret sauce for building great AI products."

"Model companies are students in a classroom wanting good points—they’re happy to run external expert evals to improve."

"The more I work with evals, the more I think UX and product people need to be involved because of the need for diverse perspectives."

Ask the Rosenbot
Alëna Iouguina
Designing Systems at Scale
2018 • DesignOps Summit 2018
Gold
John Devanney
The Design Management Office
2017 • DesignOps Summit 2017
Gold
Louis Rosenfeld
Day 1 Welcome
2024 • DesignOps Summit 2024
Gold
Maria Skaaden
Continuous Design: One eye on the horizon and the other on the next wave
2018 • DesignOps Summit 2018
Gold
Megan Nipe
Human-Centered Design for Engagement: Maturing from Newsletterville to Personalized, One-to-One Messaging
2021 • Civic Design 2021
Gold
Samuel Proulx
Designing for Disability, Innovating for Everyone
2025 • Advancing Research 2025
Gold
Llewyn Paine
[Demo] Deploying AI doppelgangers to de-identify user research recordings
2024 • Designing with AI 2024
Gold
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Noz Urbina
Rapid AI-powered UX (RAUX): A framework for empowering human designers
2025 • Rosenfeld Community
Craig Villamor
Resilient Enterprise Design
2017 • Enterprise Experience 2017
Gold
Gillian Salerno-Rebic
Redefining Speed and Scale: How Accenture’s GrowthOS Uses AI-Simulated Insights to Reduce Risk and Accelerate Innovation
2025 • Designing with AI 2025
Gold
Amy Marquez
INVEST: Discussion
2018 • Enterprise Experience 2018
Gold
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Sarah Williams
A Framework for CX Transformation
2021 • Design at Scale 2021
Gold
Sarah Brooks
Fireside chat with Sarah Brooks and Jen Pahlka
2021 • Civic Design Community
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold

More Videos

Wyatt Hayman

"The panels are primarily qualitative tools and not meant for statistically significant quantitative research."

Wyatt Hayman

Global Research Panels

August 8, 2020

Ashley Cortez

"People will remember how you make them feel, not what you say, especially when they are going through trauma."

Ashley Cortez Devika Menon Nidhi Singh Rathore Danita J. Reese Mari Nakano

Shifting Toward Community-Led Innovation in Local Government

December 8, 2021

Shahrzad Samadzadeh

"Knowing your value as a person means not giving everything to situations that don't give back."

Shahrzad Samadzadeh

What Is My Value? Two Takes and Some Mistakes

January 8, 2024

Lily Aduana

"Professional testers are very crafty; one even changed her name repeatedly to get recruited multiple times."

Lily Aduana Savannah Hobbs Brittany Rutherford

5 Reasons to Bring Your Recruiting in-House (and How To Do It)

March 12, 2021

Tim Parmee

"If you only have time for one topic in a session with VPs and heads, make it the definition of success."

Tim Parmee

Changing Our Design Pressure Points

October 2, 2023

Adam Thomas

"You have to ask, what are we deciding, and what are we quitting, on a regular basis."

Adam Thomas

Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way

December 6, 2022

Doug Powell

"This is a just-do-it moment. Ask for forgiveness, not permission."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Robert Schwartz

"We didn’t do this with arrogance. We did it with great respect, understanding who we’re delivering to, shareholders, patients, customers."

Robert Schwartz

We're Here for the Humans

June 9, 2017

Rima Campbell

"UX assets are our business assets."

Rima Campbell Amrit S Bhachu

Increase Productivity and Drive Business Impact

September 24, 2024