Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Building the Rosenbot
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Building the Rosenbot
Speakers: Peter Van Dijck
Link:

Summary

A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?

Key Insights

  • Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.

  • Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.

  • Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.

  • Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.

  • Effective evaluation requires combining human expert and end-user feedback alongside automated tools.

  • A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.

  • ‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.

  • New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.

  • UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.

  • Co-creation between users and technology defines how AI applications evolve and succeed or fail.

Notable Quotes

"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."

"This AI stuff is a new design material, just like the internet or mobile was before."

"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."

"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."

"There’s a lot of steps behind the scenes to make conversations useful and safe."

"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."

"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."

"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."

"Co-creation between users and technology determines if this AI evolves to be useful or harmful."

"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."

Ask the Rosenbot
Christian Crumlish
AMA with Christian Crumlish, author of Product Management for UX People
2022 • Enterprise Community
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps Summit 2024
Gold
Billy Carlson
Pro-level UI Tips for Beginners
2022 • DesignOps Summit 2022
Gold
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Dave Hora
A Research Skills Evolution
2021 • Advancing Research 2021
Gold
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Conference
Sam Yen
Driving Organizational Change Through Design? Do more of this and less of that
2017 • Enterprise Experience 2017
Gold
Dana Bishop
2022: The Year UX Demonstrates its Business Impact
2022 • Advancing Research 2022
Gold
Jilanna Wilson
Distributed DesignOps Management
2019 • DesignOps Community
Anna Avrekh
Expert Panel: Leading in and with Research
2022 • Advancing Research 2022
Gold
Ross Smith
Breaking Barriers with Empathy
2017 • Enterprise Experience 2017
Gold
Prayag Narula
HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer
2023 • Advancing Research 2023
Gold
Sarah Auslander
Incremental Steps to Drive Radical Innovation in Policy Design
2022 • Civic Design 2022
Gold
Greg Petroff
Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
2025 • Rosenfeld Community
Susan Simon-Daniels
War Stories LIVE! Susan Simon-Daniels
2020 • Advancing Research 2020
Gold
Bria Alexander
Theme Two Intro
2022 • DesignOps Summit 2022
Gold

More Videos

Doug Powell

"We are truly in a golden age for our profession."

Doug Powell

Closing Keynote: Design at Scale

November 8, 2018

Mila Kuznetsova

"Kids are not little adults. You can’t just simplify the text and call it for kids."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Sarah Gallimore

"Every parent we talked to had some variation of the question: this exists, can I use it right now? Where was this when I needed to apply?"

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Lada Gorlenko

"We can’t complain about not having a seat at the table if we don’t offer one of our own."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Alnie Figueroa

"It is our job to lead and transform with empathy and heart because there’s fear in everyone."

Alnie Figueroa

The Future of Design Operations: Transforming Our Craft

September 10, 2025

Landon Barnes

"Meaningfulness is linked directly to what your organization cares about in terms of goals and priorities."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Emily Eagle

"Sometimes we have to murder our darlings—let go of favorite parts to let the story or design grow."

Emily Eagle

Can't Rewind: Radio and Retail

June 3, 2019

Malini Rao

"Re-platforming is transformative not just for the product but also for the people, teams, and organization."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Asia Hoe

"Design is not just aesthetics; it’s about what the product feels like, sounds like, and how it differentiates in the market."

Asia Hoe

Partnering with Product: A Journey from Junior to Senior Design

November 29, 2023