Building new AI skills: Creating outsized UX value with evals
This video is featured in the AI Conference playlist and 1 more.
Summary
Evals are a place where UX professionals can leverage their skills in this new world, and in this talk, Peter shares how he’s used this method to help organizations create outsized value.
This talk is both practical and strategic. You will learn strategically how to position yourself or your team as key to new AI initiatives, and we will go through some hands-on skills understanding and creating evals.
Key Insights
-
•
Evals define what good looks like for AI outputs and are fundamentally UX-focused.
-
•
A typical eval contains three components: a task, a golden data set, and an evaluator.
-
•
UX professionals should start planning and building evals from day one of an AI project, not at QA or engineering stages.
-
•
Involving subject matter experts full-time and providing them simple UI boosts eval quality and iteration speed.
-
•
Generic built-in evals from tools rarely reflect project-specific quality definitions and often add little value.
-
•
Synthesized data from LLMs can help scale eval data sets but should be built upon high-quality, real data first.
-
•
LLMs can be used as fuzzy evaluators or judges to automate evaluation at scale if carefully designed.
-
•
Eval ownership and dedicated resourcing (up to 30% of project budget) are critical to success.
-
•
Evals enable UX professionals to quantitatively measure AI quality and gain influence in cross-disciplinary discussions.
-
•
The iterative eval process helps improve prompt engineering and informs better model choices quickly.
Notable Quotes
"Evals are a definition of what good looks like."
"UX people can help build the future of AI if we strap on our backpack, meaning our core skills."
"Nobody wants to do evals because it's hard work, but for UX people, it's interesting and fascinating."
"Start planning for evals on day one, it's not QA, it's an ongoing activity."
"You want your evals to run really fast so you can iterate quickly and improve the experience."
"Avoid generic built-in evals, because then you're not defining quality for your specific situation."
"Often you don't have to eval everything, just what defines the quality that matters."
"It's cognitively easier to evaluate the work somebody else did than to do it yourself. That's why LLMs can evaluate LLMs."
"We want to build confidence in quality even though AI outputs are stochastic and sometimes random."
"When some AI thing is being discussed, put up your hand and say, are we doing evals on this? My team can help."
Or choose a question:
More Videos
"The scores are a means to an end to encourage behavior change, not to create fear or just produce numbers."
Todd Healy Jess GrecoDriving Change with CX Metrics
June 7, 2023
"Some teams were going rogue or skipping research altogether."
Marjorie Stainback Kelsey KingmanTransforming Strategic Research Capacity through Democratization
October 24, 2019
"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."
Jonathan Fairman Kevin JohnsonIntegrating generative AI into enterprise products: A case study from dscout
June 5, 2024
"Questions can be weapons; language in research, especially demographic questions, can exclude and retraumatize participants."
Matt Bernius Rachael Dietkus, LCSW Aditi Joshi Alba VillamilLearnings from Applying Trauma-Informed Principles to the Research Process
March 10, 2022
"We must humanize how and what we measure and recognize as impact."
Frances YllanaTheme 2 Intro
September 24, 2024
"If you press the exact same button today and tomorrow you get two different results, that’s non-deterministic UX."
Lija Hogan Milan Mijatovic Sam Proulx Louis RosenfeldThree Years Out: Perspectives on the Near-Term Future of User Research
March 15, 2024
"Reflection in action involves noticing surprise, questioning assumptions, experimenting, and reflecting on the move."
Yoel SumitroActions and Reflections: Bridging the Skills Gap among Researchers
March 9, 2022
"The design system is a living, breathing thing that should constantly evolve based on learning and user needs."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"Nothing is inherently better because it was produced by human intention or machine learning; interrogate the goal first."
Rachael Dietkus, LCSW Llewyn Paine Nishanshi Shukla David WomackAI: Passionate defenses and reasoned critique [Advancing Research Community Workshop Series]
September 18, 2024
