Building new AI skills: Creating outsized UX value with evals
This video is featured in the AI Conference playlist and 1 more.
Summary
Evals are a place where UX professionals can leverage their skills in this new world, and in this talk, Peter shares how he’s used this method to help organizations create outsized value.
This talk is both practical and strategic. You will learn strategically how to position yourself or your team as key to new AI initiatives, and we will go through some hands-on skills understanding and creating evals.
Key Insights
-
•
Evals define what good looks like for AI outputs and are fundamentally UX-focused.
-
•
A typical eval contains three components: a task, a golden data set, and an evaluator.
-
•
UX professionals should start planning and building evals from day one of an AI project, not at QA or engineering stages.
-
•
Involving subject matter experts full-time and providing them simple UI boosts eval quality and iteration speed.
-
•
Generic built-in evals from tools rarely reflect project-specific quality definitions and often add little value.
-
•
Synthesized data from LLMs can help scale eval data sets but should be built upon high-quality, real data first.
-
•
LLMs can be used as fuzzy evaluators or judges to automate evaluation at scale if carefully designed.
-
•
Eval ownership and dedicated resourcing (up to 30% of project budget) are critical to success.
-
•
Evals enable UX professionals to quantitatively measure AI quality and gain influence in cross-disciplinary discussions.
-
•
The iterative eval process helps improve prompt engineering and informs better model choices quickly.
Notable Quotes
"Evals are a definition of what good looks like."
"UX people can help build the future of AI if we strap on our backpack, meaning our core skills."
"Nobody wants to do evals because it's hard work, but for UX people, it's interesting and fascinating."
"Start planning for evals on day one, it's not QA, it's an ongoing activity."
"You want your evals to run really fast so you can iterate quickly and improve the experience."
"Avoid generic built-in evals, because then you're not defining quality for your specific situation."
"Often you don't have to eval everything, just what defines the quality that matters."
"It's cognitively easier to evaluate the work somebody else did than to do it yourself. That's why LLMs can evaluate LLMs."
"We want to build confidence in quality even though AI outputs are stochastic and sometimes random."
"When some AI thing is being discussed, put up your hand and say, are we doing evals on this? My team can help."
Or choose a question:
More Videos
"Silence is complicity. Inaction is support."
Denise Jacobs Nancy Douyon Renee Reid Lisa WelchmanInteractive Keynote: Social Change by Design
January 8, 2024
"The prototype caught fire because the motor and axle were misaligned — my design didn’t consider the physical realities."
Dan WillisEnterprise Storytelling Sessions
June 8, 2016
"Jazz has those rules of engagement."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
June 4, 2019
"Collaboration with peers and speaker coaches creates better content and stronger relationships beyond the conference."
Louis Rosenfeld Jemma Ahmed Christian Crumlish Uday Gajendar Chris GeisonCoffee with Lou #3: What Makes for a Successful UX Conference Presentation?
May 2, 2024
"Regular surveys showed teaching quality mattered 11%, but logistic regression revealed it actually represents 77% of the decision."
Ricardo MartinsUnlocking the power of advanced quantitative methods
March 12, 2025
"Lead with the ask, lead with the value, so people understand what you’re looking for from the start."
Frank DuranPartnership Playbook: Lessons Learned in Effective Partnership
January 8, 2024
"Technology moves faster than policymaking — that’s the pacing problem we need to address."
Alexandra SchmidtWhy Ethics Can't Save Tech
November 18, 2022
"Our content strategist was a single point of failure—when she left, we scrambled to pick up unfinished work."
Davis Neable Guy SegalHow to Drive a Design Project When you Don’t Have a Design Team
June 10, 2021
"Screen readers navigate based on the code and semantics, not on visual layout or appearance."
Sam ProulxDesigning For Screen Readers: Understanding the Mental Models and Techniques of Real Users
December 10, 2021
