Building new AI skills: Creating outsized UX value with evals
This video is featured in the AI Conference playlist and 1 more.
Summary
Evals are a place where UX professionals can leverage their skills in this new world, and in this talk, Peter shares how he’s used this method to help organizations create outsized value.
This talk is both practical and strategic. You will learn strategically how to position yourself or your team as key to new AI initiatives, and we will go through some hands-on skills understanding and creating evals.
Key Insights
-
•
Evals define what good looks like for AI outputs and are fundamentally UX-focused.
-
•
A typical eval contains three components: a task, a golden data set, and an evaluator.
-
•
UX professionals should start planning and building evals from day one of an AI project, not at QA or engineering stages.
-
•
Involving subject matter experts full-time and providing them simple UI boosts eval quality and iteration speed.
-
•
Generic built-in evals from tools rarely reflect project-specific quality definitions and often add little value.
-
•
Synthesized data from LLMs can help scale eval data sets but should be built upon high-quality, real data first.
-
•
LLMs can be used as fuzzy evaluators or judges to automate evaluation at scale if carefully designed.
-
•
Eval ownership and dedicated resourcing (up to 30% of project budget) are critical to success.
-
•
Evals enable UX professionals to quantitatively measure AI quality and gain influence in cross-disciplinary discussions.
-
•
The iterative eval process helps improve prompt engineering and informs better model choices quickly.
Notable Quotes
"Evals are a definition of what good looks like."
"UX people can help build the future of AI if we strap on our backpack, meaning our core skills."
"Nobody wants to do evals because it's hard work, but for UX people, it's interesting and fascinating."
"Start planning for evals on day one, it's not QA, it's an ongoing activity."
"You want your evals to run really fast so you can iterate quickly and improve the experience."
"Avoid generic built-in evals, because then you're not defining quality for your specific situation."
"Often you don't have to eval everything, just what defines the quality that matters."
"It's cognitively easier to evaluate the work somebody else did than to do it yourself. That's why LLMs can evaluate LLMs."
"We want to build confidence in quality even though AI outputs are stochastic and sometimes random."
"When some AI thing is being discussed, put up your hand and say, are we doing evals on this? My team can help."
Or choose a question:
More Videos
"Creativity fuels design, design fuels innovation."
Dr. Karl JeffriesThe Science of Creativity for DesignOps
January 8, 2024
"You have to understand how the business actually makes money to align UX with it."
Amy MarquezINVEST: Discussion
June 15, 2018
"Trust is built by showing you’re there to help make them succeed, not just push your agenda."
Steve Portigal Chris Chapo Kelly Goto Christian RohrerDiscussion
May 13, 2015
"Agency style staffing allows designers to bring learnings across multiple projects and grow through new challenges."
Alicia MootyDesign Staffing Models
September 30, 2021
"UX should be a critical dependency, not an afterthought or checkbox on the to-do list."
Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt LaurenceThe Authentic UX Talent Show
January 8, 2024
"Operations tends to be something companies only think about when they are scaling rapidly, but investing early is essential."
Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne RogondinoTheme 1: Discussion
January 8, 2024
"We think about research as the magnetic field that attracts every part of the business into alignment around the customer."
Sofia QuinteroThe Product Philosophy Behind EnjoyHQ
March 10, 2021
"Please don’t chop off that engine until the new one is built because you are not the only one on this plane."
Laura KleinUnique challenges of innovation in enterprises
April 23, 2020
"At Hotcho, we started with a qualitative focus, swung too far into quantitative, and then found a balanced middle ground."
Andrew MichaelBuilding a Product Insights Team
March 10, 2022
