This video is featured in the AI and UX playlist.
Summary
This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"We want to deepen equity and care in how we work, who we do it with, and the output and impact it generates."
Ariel KennanTheme Two Intro
November 17, 2022
"Without automation, none of these tools are talking to each other, and operational overhead for your Ops and DPM teams is actually absurd."
Candace MyersStandardizing Design at Scale
September 9, 2022
"There’s a tension between showing urgency emotionally and delivering detailed data practically in one product."
Marissa Cui Rachel He Michael Leggett Manos SaratsisClimate Design Product Showcase
March 13, 2024
"Our research practice was perceived largely as design testers, empathy vehicles, and policing functions — a narrow, reactive role."
Nalini KotamrajuResearch After UX
March 25, 2024
"With enough creativity and imagination, you can make anything interesting."
Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt LaurenceThe Authentic UX Talent Show
January 8, 2024
"Establish yourself as a trusted expert advisor whose work drives meaningful positive change."
Megan BlockerGetting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
March 26, 2024
"Do you have personas for the people that work for you? Do you use them to help define how your processes work?"
Holly ColeUnderstanding Experiences: When you have to do more than work
November 8, 2018
"Design systems allow us to wrap up a prototype quickly, test, fail fast, and iterate."
Scher Foord Corey Greenltch Sarah RoweTurn the Ship Around: How to Apply Design Thinking Across Your Organization
June 10, 2021
"I literally ask, how are you feeling right now? It’s about connecting on a human level before diving into data."
Jerome “Axle” BrownHow to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon
March 11, 2021
