Rosenverse
Hands-on AI #2: Understanding evals: LLM as a Judge

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Hands-on AI #2: Understanding evals: LLM as a Judge

Wednesday, October 15, 2025 • Rosenfeld Community

This video is featured in the Evals + Claude playlist.

Share the love for this talk
Hands-on AI #2: Understanding evals: LLM as a Judge
Speakers: Peter Van Dijck
Link:

Summary

If you’re a product manager, UX researcher, or any kind of designer involved in creating an AI product or feature, you need to understand evals. And a great way to learn is with a hands-on example.

In this second talk in the series, Peter Van Dijck of the helpful intelligence company will show you how to create an eval for an AI product using an LLM as a judge (when we use a Large Language Model to evaluate the output of another Large Language Model). We’ll have a look at how that works, but also dig into why this even works. Are we creating problems for ourselves when we let an LLM judge itself? This talk is hands on; and there will be plenty of time for questions.

You will go away understanding when and how to use LLM as a judge, and build some product sense around how the best AI products today are built, and how that can help you use them more effectively yourself.

Key Insights

  • •

    Evals are a foundational feedback loop defining what 'good' means for AI products, helping to measure and improve systems continuously.

  • •

    Evaluating fuzzy, subjective AI outputs requires innovative approaches such as using LLMs as judges to score results.

  • •

    Binary (yes/no) scoring is more reliable than rating scales with ranges because LLMs lack internal memory and consistency.

  • •

    Starting evals early (week one of a project) drastically improves AI product outcomes, but many teams delay due to perceived complexity.

  • •

    High-risk or important tasks should be prioritized for evals instead of attempting broad coverage.

  • •

    Assigning a dedicated owner or 'benevolent dictator' for evals who works closely with domain experts accelerates feedback and quality.

  • •

    Creating a written constitution of principles helps concretize AI behavior goals and guides prompt and model training.

  • •

    Most current eval tooling is too technical, slowing iteration cycles and making expert involvement inefficient.

  • •

    Custom feedback interfaces tailored to expert users significantly speed up evaluating AI outputs in domains like healthcare and law.

  • •

    Diverse perspectives from UX, product, strategy, and domain experts are critical in defining and refining what 'good' means in AI systems.

Notable Quotes

"Evals are everywhere, right? Everybody's talking about evals. It is like one of the key things in developing useful AI products."

"You want to ask an LLM to evaluate the fuzzy stuff because there’s no black and white output."

"LLMs don’t have memory, so rating on a scale from one to five is pretty random. Better to have yes or no answers."

"One of the biggest problems in AI building is evolving your prompts and having a fast feedback loop."

"By starting to categorize risk in detail, you naturally lead to better prompts and better evals."

"A constitution is a very good exercise: write down your system’s principles and values to help guide its behavior."

"Use custom systems for experts to quickly review and rate outputs, making feedback cycles much faster."

"Evals define a shared definition of good with tests to measure it, and that is the secret sauce for building great AI products."

"Model companies are students in a classroom wanting good points—they’re happy to run external expert evals to improve."

"The more I work with evals, the more I think UX and product people need to be involved because of the need for diverse perspectives."

Ask the Rosenbot
Spencer L. A. Stultz
Why Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives
2023 • DesignOps Summit 2023
Gold
Fredrik Matheson
First-time users, longtime strategies: Why Parkinson’s Law is making you less effective at work – and how to design a fix.
2016 • Enterprise UX 2016
Gold
Abby Covert
Is the shift greater than the sum of its parts?
2026 • Shift UX 2026
Conference
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Aleksandra Korczynska
Survey Tools
2026 • Advancing Research 2026
Gold
Andrew Webster
Scaling Design Capability: How Involved Should You Be?
2021 • DesignOps Summit 2021
Gold
Liwei Dai
The Heart and Brain of the AI Research
2020 • Advancing Research 2020
Gold
Jennifer Fraser
What would Emmy Noether Do? Math, Models and Mulling in UX Research
2023 • Advancing Research 2023
Gold
Jon White
Unsticking Research for Better Information Flow
2026 • Advancing Research 2026
Gold
Corey Nelson
Layoffs
2022 • Advancing Research Community
Brigette Metzler
Scaling ResearchOps: Helping Researchers do Their Best Work
2020 • Advancing Research 2020
Gold
Bria Alexander
Opening Remarks
2021 • Design at Scale 2021
Gold
Nathan Shedroff
Double Your Mileage: Use Your Research Strategically
2020 • Advancing Research 2020
Gold
Carl Turner
You Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
2023 • Advancing Research 2023
Gold
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Dave Gray
Liminal Thinking: Sense-making for systems in large organizations
2015 • Enterprise UX 2015
Gold

More Videos

Matt Webb

"Computers got ten years better overnight with the arrival of GPT-3 in November 2022."

Matt Webb

Context Window: Five Futures for AI

June 11, 2025

Sofia Quintero

"It’s about helping your colleagues be more productive and make better decisions to help customers."

Sofia Quintero

Beyond Tools: The Messy Business of Implementing Research Repositories

March 10, 2022

Mansi Gupta

"Separating genders in research can perpetuate stereotypes rather than challenge them."

Mansi Gupta

Advancing the Inclusion of Womxn in Research Practices

September 15, 2022

Matteo Gratton

"Data per se are not the problem; it is the way we are using the data that can be problematic."

Matteo Gratton

Can Data and Ethics Live Together?

October 1, 2021

Tara Tressel

"People were more likely to explain their thought process and just more context around the particular situation of their org."

Tara Tressel

Investigating qualitative depth of AI-moderated interviews

March 10, 2026

Louis Rosenfeld

"Climate UX feels like 1995 again, an exciting time of discovery and cross-domain jumps."

Louis Rosenfeld Matt Jones Olga Khroustaleva Michael Leggett Karol Munoz

Do you want to work on climate? (Climate UX Discussion Series)

November 15, 2023

David Cronin

"A lot of people were just going and downloading the tear sheet, which is to be expected."

David Cronin

The GE Design System and Thoughts about Craft at Scale

May 13, 2015

Laureen Kattan

"We decided not to build a new app but instead integrate within the legacy electronic health record system."

Laureen Kattan Julie Kim

Centering Patients and Clinicians in a Complex Government Ecosystem

November 29, 2023

Michael Weir

"If a website is hard to use, motivation and approach drops, and users are less likely to engage or convert."

Michael Weir

Mixed Methods and Behavioural Science

May 26, 2023

Latest Books All books