Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Building new AI skills: Creating outsized UX value with evals
This video is featured in the AI Conference playlist and 1 more.
Summary
AI is dramatically changing how engineering works (code is cheap now), and UX people of all sorts are figuring out how they fit in this new way of building. Evals are a place where UX professionals can leverage their skills in this new world, and in this talk, Peter shares how he’s used this method to help organizations create outsized value. This talk is both practical and strategic. You will learn strategically how to position yourself or your team as key to new AI initiatives, and we will go through some hands-on skills understanding and creating evals.
Key Insights
-
•
Evals define what good looks like for AI outputs and are fundamentally UX-focused.
-
•
A typical eval contains three components: a task, a golden data set, and an evaluator.
-
•
UX professionals should start planning and building evals from day one of an AI project, not at QA or engineering stages.
-
•
Involving subject matter experts full-time and providing them simple UI boosts eval quality and iteration speed.
-
•
Generic built-in evals from tools rarely reflect project-specific quality definitions and often add little value.
-
•
Synthesized data from LLMs can help scale eval data sets but should be built upon high-quality, real data first.
-
•
LLMs can be used as fuzzy evaluators or judges to automate evaluation at scale if carefully designed.
-
•
Eval ownership and dedicated resourcing (up to 30% of project budget) are critical to success.
-
•
Evals enable UX professionals to quantitatively measure AI quality and gain influence in cross-disciplinary discussions.
-
•
The iterative eval process helps improve prompt engineering and informs better model choices quickly.
Notable Quotes
"Evals are a definition of what good looks like."
"UX people can help build the future of AI if we strap on our backpack, meaning our core skills."
"Nobody wants to do evals because it's hard work, but for UX people, it's interesting and fascinating."
"Start planning for evals on day one, it's not QA, it's an ongoing activity."
"You want your evals to run really fast so you can iterate quickly and improve the experience."
"Avoid generic built-in evals, because then you're not defining quality for your specific situation."
"Often you don't have to eval everything, just what defines the quality that matters."
"It's cognitively easier to evaluate the work somebody else did than to do it yourself. That's why LLMs can evaluate LLMs."
"We want to build confidence in quality even though AI outputs are stochastic and sometimes random."
"When some AI thing is being discussed, put up your hand and say, are we doing evals on this? My team can help."
Or choose a question:
More Videos
"Touch screens let blind users use physical spatial information in a way desktops never could."
Sam ProulxUnderstanding Screen Readers on Mobile: How And Why to Learn from Native Users
March 27, 2023
"The path of the eagle is technological growth without wisdom, while the condor represents spiritual respect for life."
Jayne Engle Tanya Chung-Tiam-FookCivic Design for the Next Seven Generations—A Discussion on Sacred Civics
August 25, 2022
"You can’t evaluate bias if the AI can’t explain itself."
Daniel J. RosenbergDesigning with and for Artificial Intelligence
August 11, 2022
"AI agents take all your research and instantly synthesize briefs and code; the feedback loop is real-time and continuous."
Erika FlowersThe Handoff is Dead: Design-Led Engineering with AI Agents
March 4, 2026
"Machines can identify correlation via association, but it’s extremely difficult for them to reason and identify causality."
Liwei DaiThe Heart and Brain of the AI Research
March 31, 2020
"Nobody’s going to use digital calendars because they’re just more work and no benefit."
Sam LadnerData Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX
June 8, 2016
"You have to understand how the business actually makes money to align UX with it."
Amy MarquezINVEST: Discussion
June 15, 2018
"Leadership is not about having all the answers, it’s about empowering others to own the strategy."
Jacqui FreyFlow and Superfluidity for Design Orgs
November 7, 2018
"The stories we tell are framed by the questions we ask. Our stories repeated become our beliefs and our beliefs influence our behavior."
Brendan JarvisFraming Tomorrow by Questioning Today
June 8, 2022
