Designing AI-first products on top of a rapidly evolving technology
This video is featured in the AI and UX playlist.
Summary
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"Silence is complicity. Inaction is support."
Denise Jacobs Nancy Douyon Renee Reid Lisa WelchmanInteractive Keynote: Social Change by Design
January 8, 2024
"When you’re new and inexperienced, asking for help feels like admitting you don’t belong, but really, it shows wisdom."
Dan WillisEnterprise Storytelling Sessions
June 8, 2016
"Within improv, it’s a combination of listening and not trying to be funny."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
June 4, 2019
"If your talk isn’t accepted, it doesn’t mean it’s bad—it might just not fit the program’s narrative arc or available slots."
Louis Rosenfeld Jemma Ahmed Christian Crumlish Uday Gajendar Chris GeisonCoffee with Lou #3: What Makes for a Successful UX Conference Presentation?
May 2, 2024
"Regular surveys showed teaching quality mattered 11%, but logistic regression revealed it actually represents 77% of the decision."
Ricardo MartinsUnlocking the power of advanced quantitative methods
March 12, 2025
"I’d argue you couldn’t be effective in any role without some sense of how to create partnerships and move forward."
Frank DuranPartnership Playbook: Lessons Learned in Effective Partnership
January 8, 2024
"I’m totally freaked out by facial recognition and wonder why we don’t get together as consumers to push back on bad designs."
Alexandra SchmidtWhy Ethics Can't Save Tech
November 18, 2022
"The complex problem we were solving was not really solution based at all. It was a cultural team alignment problem."
Davis Neable Guy SegalHow to Drive a Design Project When you Don’t Have a Design Team
June 10, 2021
"If you’re designing a site used infrequently, don’t overload it with custom hotkeys that are hard to memorize."
Sam ProulxDesigning For Screen Readers: Understanding the Mental Models and Techniques of Real Users
December 10, 2021
Latest Books All books
Dig deeper with the Rosenbot
What are AI skills and how should their names and descriptions be structured for better findability?
What are practical ways to split accent colors for accessibility without breaking brand recognition?
How do current pressures like AI tools and offshoring affect designers’ sense of purpose and warmth in design?