Rosenverse
Designing AI-first products on top of a rapidly evolving technology

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Designing AI-first products on top of a rapidly evolving technology

Gold
Tuesday, June 10, 2025 • Designing with AI 2025

This video is featured in the AI and UX playlist.

Share the love for this talk
Designing AI-first products on top of a rapidly evolving technology
Speakers: Peter Van Dijck
Link:

Summary

This talk addresses the distinct challenges of designing AI-first products when the foundational technology is advancing at an unprecedented pace. We will discuss the practical implications of this rapid evolution on the design process and share our insights into creating effective user experiences in such a dynamic environment.

Key Insights

  • AI models evolve faster than previous tech revolutions, requiring adaptive product development.

  • Effective AI products use evals—automated tests on real data—to continuously measure quality.

  • Retrieval augmented generation and tool use are practical architectural tricks to reduce AI hallucination.

  • AI experiences must account for user trust despite AI’s confident but sometimes inaccurate outputs.

  • Human factors like worker anxiety about job security are critical to address in AI adoption.

  • Cross-disciplinary teams are essential in AI design due to the space’s novelty and uncertainty.

  • Prompt engineering skills grow best through extensive real use rather than frozen techniques.

  • Planning AI products with expectations of improved capabilities 6-9 months ahead optimizes outcomes.

  • Cultural acceptance of AI use inside organizations must be actively encouraged to avoid user hiding.

  • User testing of AI products still requires old-school observation combined with analytics and evals.

Notable Quotes

"You can’t have good AI without good information architecture."

"Everything’s changing really quickly—if you don’t revise your plan in a few months, you’re building for a world that’s already outdated."

"Hallucinations are when AI makes up stuff—it's a fundamental property of these models."

"Retrieval augmented generation is just adding results from a search into the model’s context to make it more grounded."

"Agents are models using tools in the loop, calling normal software functions as part of their process."

"People tend to trust AI a lot because it seems very confident—even when it’s not always accurate."

"Workers are often worried about layoffs and their value when AI speeds up their process, while executives like the efficiency gains."

"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."

"Building evals is like building tests to know if your AI system is good enough, and you can rerun them as models improve."

"User testing for AI still means watching people use it, plus collecting analytics and running evaluations."

Ask the Rosenbot
Jon Fukuda
Theme 3 Intro
2024 • DesignOps Summit 2024
Gold
Sarah Auslander
Insights Panel
2022 • Civic Design 2022
Gold
Anne Cantera
The New Design Stack: The Skills Traditional Designers Need to Add To Their Toolboxes ASAP
2026 • Rosenfeld Community
Megan Blocker
Theme 2 Intro
2025 • Advancing Research 2025
Gold
Nathan Shedroff
How Will Design be Taught When the Schools Shut Down?
2026 • Rosenfeld Community
Steve Portigal
Discussion
2015 • Enterprise UX 2015
Gold
Dave Malouf
Panel: Design Systems and Documentation
2017 • DesignOps Summit 2017
Gold
Lada Gorlenko
Theme 1: Intro
2024 • Enterprise Experience 2020
Gold
Theresa Neil
Designing for Wellness: Specializing in Healthcare
2024 • Rosenfeld Community
Matt Stone
Scaling Empathy, A Case Study in Change Management
2021 • Design at Scale 2021
Gold
Jane Reid
Self-care in User Research
2020 • Advancing Research Community
Nathan Shedroff
Double Your Mileage: Use Your Research Strategically
2020 • Advancing Research 2020
Gold
Crystal Philcox
The Many Faces of Operations
2017 • DesignOps Summit 2017
Gold
How to Identify and Increase your "Experience Quotient"
2018 • Enterprise Experience 2018
Gold
Alana Washington
Theme 3 Intro
2021 • DesignOps Summit 2021
Gold
Rebecca Buck
Mission: Keep Talent in Research Roles!
2021 • Advancing Research 2021
Gold

More Videos

Sara Conklin

"I started at absolute zero with climate knowledge and now I have clarity, confidence, credibility, and ability to contribute."

Sara Conklin

A UXer’s 12-Month Journey from Climate Concern to Climate Credibility

June 26, 2025

Benjamin Real

"We didn’t explain the philosophy behind the design ops activities, so others didn’t understand why some activities couldn’t be done outside design."

Benjamin Real

Showing the Value of DesignOps by Not Having a DesignOps Team

October 21, 2020

Lena Shenkarenko

"We must create spaces that allow nature to thrive within our cities."

Lena Shenkarenko

Collaborative Wireframing for Creating Team Alignment and Shipping Better Products

October 21, 2020

Maria Taylor

"Building trust should absolutely be a core value for design ops teams aiming to steward knowledge effectively."

Maria Taylor

Knowledge is Power: Managing the Lifeblood of the Design Org

October 3, 2023

Liz Ebengo

"You begin research from the end—from the future to the future—asking communities to share their envisioned future first."

Liz Ebengo

The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward

December 4, 2024

Roberta Dombrowski

"Sharing small, incremental updates about research progress provides visibility and makes people appreciate the craft behind the work."

Roberta Dombrowski Sam Duong Woloszynski

Making Research a Team Sport

March 11, 2022

Rikki Teeters

"At Anthropic, 80% of the code merged to production is now written by Claude."

Rikki Teeters

Concept to code: Transforming ideas into functional products with Kiro

June 10, 2026

Saara Kamppari-Miller

"Look for the commonality in our definition of inclusion rather than the differences."

Saara Kamppari-Miller

Theme Three Intro

September 9, 2022

Jorge Arango

"Handwritten notes have no boot up time, and that immediacy helps enter deeper flow states."

Jorge Arango

The Best of Both Worlds: How to Integrate Paper and Digital Notes (1st of 3 seminars)

April 5, 2024