Designing AI-first products on top of a rapidly evolving technology
This video is featured in the AI and UX playlist.
Summary
Key Insights
-
•
AI models evolve faster than previous tech revolutions, requiring adaptive product development.
-
•
Effective AI products use evals—automated tests on real data—to continuously measure quality.
-
•
Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.
-
•
AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.
-
•
Human factors like worker anxiety about job security are critical to address in AI adoption.
-
•
Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.
-
•
Prompt engineering skills grow best through extensive real use rather than frozen techniques.
-
•
Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.
-
•
Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.
-
•
User testing of AI products still requires old-school observation combined with analytics and evals.
Notable Quotes
"You can’t have good AI without good information architecture."
"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."
"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."
"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."
"Agents are models using tools in the loop, calling normal software functions as part of their process."
"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."
"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."
"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."
"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."
"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."
Or choose a question:
More Videos
"Personalization of accessibility settings will be the next big shift, letting users avoid repeated microaggressions like repeatedly enabling captions."
Sheri Byrne-HaberThe Importance of Accessible Design Systems
January 8, 2024
"It takes roughly one and a half times more investment to build reusable widgets, so we aim for a minimum of three adoptions to ensure ROI."
Dawn ResselFull-Stack User Experiences: A Marriage of Design and Technology
June 9, 2016
"Product operations serve as the binding agent to enable teams and processes to work seamlessly together."
Sean McKayWhole Product Thinking: Expanding beyond problem and solution space thinking
March 14, 2024
"There are too many things all over the place. I would like to use my magic wand for making them one system."
Abby Covert Tomer SharonPanel: Collaboration Tools
November 6, 2017
"The LAUSD stakeholder felt armed with data and designs to articulate future benefits in a language the school board could understand and quantify."
JD BuckleyCommunicating the ROI of UX within a large enterprise and out on the streets
June 14, 2018
"Building strong relationships with marketing requires mutual respect and regularly following up on their insights."
Joanna Vodopivec Prabhas PokharelOne Research Team for All - Influence Without Authority
March 9, 2022
"Listening with all your senses, not just the words they’re saying, reveals people’s true needs."
Melissa Eggleston Maya Israni Florence Kasule Owen Seely Andrea SchneiderPractical People Skills for Building Trust on Teams and with Partners
December 9, 2021
"As a young Black man with long hair entering traditional civic spaces, I had to prove my authority through support from leaders like the mayor."
Justin Entzminger Terrance Smith Tracy M. Colunga Mai-Ling GarciaRisk and Reward: How to Diversify the Field of Civic Innovators and Designers
November 17, 2022
"Your company is a service provider; your HR, IT, design, and security teams are all service providers."
Jeff SussnaWhat DesignOps Can Learn From DevOps
November 6, 2017
