This video is featured in the AI and UX playlist.
Summary
This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"It takes roughly one and a half times more investment to build reusable widgets, so we aim for a minimum of three adoptions to ensure ROI."
Dawn ResselFull-Stack User Experiences: A Marriage of Design and Technology
June 9, 2016
"Right here problems are the questions your PM asks you to fix this quarter to move forward."
Peter LevinSolve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility
March 25, 2024
"Disability isn’t a fixed group — aging, illness, injury can put anyone into a disabled state."
Gabriela BarnevaOperationalizing Inclusive Design in Service Design
November 20, 2025
"The ultimate goal is that feedback leads to actual actions, whether quick wins or longer strategic roadmap changes."
Anna Nguyen Emily BroganWhy Our Voice of the Customer is Better Than Yours
March 10, 2022
"I feel like I helped to create the environment that that came from, so I certainly have responsibility."
Jim KalbachPeace is waged with sticky notes: Mapping Real-World Experiences
June 14, 2018
"Service lines bridge gaps between product lines by facilitating knowledge flow and alignment."
Alëna IouguinaDesigning Systems at Scale
November 7, 2018
"The transition to work from home went really well and employees felt supported with flexible schedules and wellness allowances."
Kurdin Bazaz Liz Rytting Alex KarrCulture, DIBS & Recruiting
June 10, 2021
"Assistive technology can feel like another person in the room during a research session, which you have to understand and include."
Kavana RameshMeaningful inclusion: Practicing accessibility research with confidence
September 24, 2024
"Knowledge gardening requires continuous nurturing — revisiting notes, expanding them over time, not just capturing and forgetting."
Jorge ArangoExploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)
April 19, 2024
Latest Books All books
Dig deeper with the Rosenbot
How might quantitative ethnography enable real-time qualitative insights during ongoing data collection?
How do research operations teams manage the tension between consolidating tools and needing specialized capabilities?
How can AI help automate the population of a UX research repository from unstructured data sources?