This video is featured in the AI and UX playlist.
Summary
This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"You don’t want to give too many options like rating from one to ten because consistency gets lost between different LLM calls."
Peter Van DijckBuilding impactful AI products for design and product leaders, Part 2: Evals are your moat
July 23, 2025
"AI innovation has a very small margin of success—80% of AI products fail before market, and 40% of those that launch never succeed."
Jodi ForlizziDesign and AI innovation
June 5, 2024
"The introvert’s edge is that they research, plan, and have a strategy for meaningful conversations."
Dave Hoffer Joanne WeaverUX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
May 21, 2025
"The proof of service design’s effectiveness is now out there in the growing diversity of case studies and practitioner voices."
Lavrans Løvlie Ben ReasonAsk me anything – Authors of Service Design: From Insight to Implementation
November 19, 2025
"Providing stakeholders with real user voices alongside insights is key to internalizing and acting on research."
Andy Barraclough Betsy NelsonFrom Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
September 23, 2024
"I didn’t understand just how complex everything was both on the stakeholder and technology side."
Louis Rosenfeld Lashanda Hodge Senongo Akpem Chris HodowanecBecoming a Civic Designer: Making the Move from Private to Public Sector
November 17, 2022
"Why was the universe doing this to me? That was a bad question."
Brendan JarvisFraming Tomorrow by Questioning Today
June 8, 2022
"We developed family-facing intake tools to make the service experience more transparent and equitable."
Shanti Mathew Natalie Sims Natalia RadywylCivic Design at Scale: Introducing the Public Policy Layer Cake
December 9, 2021
"Mastery is achievable, whether you have 10,000 hours or not—it’s about building that portfolio and case studies."
Liam ThurstonWhy Your Design Team Is Quitting, And How To Fix It
June 10, 2022
