Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Building new AI skills: Creating outsized UX value with evals
This video is featured in the AI and UX skill growth playlist.
Summary
AI is dramatically changing how engineering works (code is cheap now), and UX people of all sorts are figuring out how they fit in this new way of building. Evals are a place where UX professionals can leverage their skills in this new world, and in this talk, Peter shares how he’s used this method to help organizations create outsized value. This talk is both practical and strategic. You will learn strategically how to position yourself or your team as key to new AI initiatives, and we will go through some hands-on skills understanding and creating evals.
Key Insights
-
•
Evals define what good looks like for AI outputs and are fundamentally UX-focused.
-
•
A typical eval contains three components: a task, a golden data set, and an evaluator.
-
•
UX professionals should start planning and building evals from day one of an AI project, not at QA or engineering stages.
-
•
Involving subject matter experts full-time and providing them simple UI boosts eval quality and iteration speed.
-
•
Generic built-in evals from tools rarely reflect project-specific quality definitions and often add little value.
-
•
Synthesized data from LLMs can help scale eval data sets but should be built upon high-quality, real data first.
-
•
LLMs can be used as fuzzy evaluators or judges to automate evaluation at scale if carefully designed.
-
•
Eval ownership and dedicated resourcing (up to 30% of project budget) are critical to success.
-
•
Evals enable UX professionals to quantitatively measure AI quality and gain influence in cross-disciplinary discussions.
-
•
The iterative eval process helps improve prompt engineering and informs better model choices quickly.
Notable Quotes
"Evals are a definition of what good looks like."
"UX people can help build the future of AI if we strap on our backpack, meaning our core skills."
"Nobody wants to do evals because it's hard work, but for UX people, it's interesting and fascinating."
"Start planning for evals on day one, it's not QA, it's an ongoing activity."
"You want your evals to run really fast so you can iterate quickly and improve the experience."
"Avoid generic built-in evals, because then you're not defining quality for your specific situation."
"Often you don't have to eval everything, just what defines the quality that matters."
"It's cognitively easier to evaluate the work somebody else did than to do it yourself. That's why LLMs can evaluate LLMs."
"We want to build confidence in quality even though AI outputs are stochastic and sometimes random."
"When some AI thing is being discussed, put up your hand and say, are we doing evals on this? My team can help."
Or choose a question:
More Videos
"Making space for subculture to thrive is like letting more air and sunlight in, it creates a healthier environment for everyone."
Spencer L. A. StultzWhy Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives
October 4, 2023
"If you don’t clarify the roles and who does what, it’s kind of impossible to do anything else."
Louis Rosenfeld Christian CrumlishOpening Remarks
November 29, 2023
"Please read our code of conduct — even though we’re virtual, we are a community obligated to kindness and respect."
Bria AlexanderOpening Remarks
November 18, 2022
"We had 80 designers attending UX Weekly and trying to get through 64 slides in one hour. It wasn’t sustainable."
Briana Thomas Christina RodriguezWhen Design Ops Comes in H.O.T. : A Tale of a Transformed Design Org
September 30, 2021
"Without shared ownership and participation, design systems risk becoming isolated and irrelevant."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"If you cannot remove a treatment, it may not be a true cause."
Joshua NobleCasual Inference
October 6, 2023
"Understanding is not a scientific process, it’s relational, personal, human, and qualitative."
Robert FabricantShifting dynamics: The evolving relationship between researchers, participants, and organizational systems
March 11, 2025
"White savior mentality is something very humbling to experience and work through."
Dolly ParikhExit Interview #7: Journey of a Social Entrepreneur
May 21, 2026
"I am so glad to be part of this new design ops community and wish you all the best of success."
Stephen PollardClosing Keynote: Getting giants to dance - what can we learn from designing large and complex public infrastructure?
November 7, 2017
