Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Designing AI-first products on top of a rapidly evolving technology

Gold
Tuesday, June 10, 2025 • Designing with AI 2025

This video is featured in the AI and UX playlist.

Share the love for this talk
Designing AI-first products on top of a rapidly evolving technology
Speakers: Peter Van Dijck
Link:

Summary

This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.

Key Insights

  • AI models evolve faster than previous tech revolutions, requiring adaptive product development.

  • Effective AI products use evals—automated tests on real data—to continuously measure quality.

  • Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.

  • AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.

  • Human factors like worker anxiety about job security are critical to address in AI adoption.

  • Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.

  • Prompt engineering skills grow best through extensive real use rather than frozen techniques.

  • Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.

  • Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.

  • User testing of AI products still requires old-school observation combined with analytics and evals.

Notable Quotes

"You can’t have good AI without good information architecture."

"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."

"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."

"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."

"Agents are models using tools in the loop, calling normal software functions as part of their process."

"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."

"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."

"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."

"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."

"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."

Ask the Rosenbot
Steve Baty
Discussion
2016 • Enterprise UX 2016
Gold
Maria Giudice
Becoming a Changemaker by Leading with Design
2023 • Advancing Research 2023
Gold
Nathan Curtis
Beyond the Toolkit: Spreading a System Across People & Products
2016 • Enterprise UX 2016
Gold
Sheryl Cababa
Day 2 Panel
2024 • Designing with AI 2024
Gold
Bria Alexander
Welcome
2022 • DesignOps Summit 2022
Gold
Andy Barraclough
From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme
2024 • DesignOps Summit 2024
Gold
Etienne Fang
The Power of Care: From Human-Centered Research to Humanity-Centered Leadership
2021 • Advancing Research 2021
Gold
Jennifer Strickland
Fireside Chat: How Design Addresses a World on Fire
2022 • Civic Design Community
Dr. Karl Jeffries
The Science of Creativity for DesignOps
2024 • DesignOps Summit 2020
Gold
Changying (Z) Zheng
Practical DesignOps: From Ideas to Tools That Teams Actually Use
2025 • Rosenfeld Community
Lisa Gironda
Opener: Chief of Staff–An unexpected journey
2024 • DesignOps Summit 2020
Gold
Sarah Williams
Verizon_A Framework for CX Transformation
2024 • Design at Scale 2021
Gold
Marc Rettig
The Urge to Migrate
2026 • Rosenfeld Community
Robin Beers
Research as a Catalyst for Organizational Transformation
2021 • Advancing Research 2021
Gold
Megan Blocker
Getting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
2024 • Advancing Research 2024
Gold
Louis Rosenfeld
Opening Remarks
2023 • DesignOps Summit 2023
Gold

More Videos

Mariah Hay

"Technology is easy, people are not dealing with people, it takes a lot out of everyone."

Mariah Hay Marina Martin Husani Oakley Eduardo Ortiz

BUILD: Discussion

June 14, 2018

Theresa Marwah

"We had 24 different consent scenarios but wanted to avoid a 15-page form that scares participants."

Theresa Marwah

How Atlassian is Operationalizing Respect in Research

February 27, 2020

Bria Alexander

"We’ve got you covered with notes, sketch notes, slides, and recordings so you can sit back, relax, and enjoy the show."

Bria Alexander Louis Rosenfeld

Opening Remarks Day 1

March 25, 2024

Kit Unger

"The past couple of years have been intense with one crisis after another."

Kit Unger

Theme 1 Intro

June 8, 2022

Eric Shumake

"It’s easier to sell if the team is talking about the tool — popularity inside counts."

Eric Shumake

An AMA on UX's Role in Healthcare

April 22, 2026

Randolph Duke II

"I put down my question guide and just talked with Carol."

Randolph Duke II

War Stories LIVE! Randy Duke II

March 30, 2020

Christopher Noessel

"AI should augment humans, not replace them, especially given societal disparities between deep and fast thinkers."

Christopher Noessel

AI of the now: Designing for Agents

July 31, 2024

Milan Guenther

"Blaming the system for failures is like blaming the garden for not growing—the system is made by the people within it."

Milan Guenther

A Shared Language for Co-Creating Ambitious Endeavours

June 6, 2023

Alla Weinberg

"Psychological safety is the foundation upon which collaboration, innovation, and performance are built."

Alla Weinberg

People Are Sick of Change: Psychological Safety is the Cure

July 20, 2023