Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Building the Rosenbot

Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Building the Rosenbot
Speakers: Peter Van Dijck
Link:

Summary

A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?

Key Insights

  • Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.

  • Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.

  • Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.

  • Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.

  • Effective evaluation requires combining human expert and end-user feedback alongside automated tools.

  • A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.

  • ‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.

  • New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.

  • UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.

  • Co-creation between users and technology defines how AI applications evolve and succeed or fail.

Notable Quotes

"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."

"This AI stuff is a new design material, just like the internet or mobile was before."

"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."

"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."

"There’s a lot of steps behind the scenes to make conversations useful and safe."

"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."

"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."

"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."

"Co-creation between users and technology determines if this AI evolves to be useful or harmful."

"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."

Ask the Rosenbot
Neil Barrie
Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
2024 • Advancing Research 2024
Gold
Joel Branch
Humanizing AI: Filling the Gaps with Multi-faceted Research
2021 • Advancing Research 2021
Gold
Anupama Dhareshwar
From blueprint to bot: Designing resilient AI-powered services
2025 • Advancing Service Design 2025
Gold
Jeff Gothelf
Who does what by how much?
2025 • Advancing Service Design 2025
Gold
Bria Alexander
Theme Two Intro
2023 • DesignOps Summit 2023
Gold
Margot Bloomstein
Fostering Trust in Your Brand and Beyond
2020 • Enterprise Community
Sylvie Abookire
A Civic Designer's Guide to Mindful Conflict Navigation
2022 • Civic Design 2022
Gold
Rachel Posman
A Closer Look at Team Ops and Product Ops (Two Sides of the DesignOps Coin)
2020 • DesignOps Community
John Cutler
Oxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)
2024 • Advancing Service Design 2024
Gold
Maverick Chan
From Doodle to Demo: AI as Our Storytelling Partner
2025 • Rosenfeld Community
Elena Naids
The Power of Difficult Conversations: A Case Study on How We Introduced Design Ops in the Federal Government Space
2023 • DesignOps Summit 2023
Gold
Matt Duignan
Atomizing Research: Trend or Trap
2020 • Advancing Research 2020
Gold
Dave Malouf
Panel: Design Systems and Documentation
2017 • DesignOps Summit 2017
Gold
Uday Gajendar
The Rise of Meta-Design: A Starter Playbook
2022 • Enterprise Community
Jamie Beck Alexander
How can you find your role in climate?
2024 • Climate UX Interest Group
Kate Koch
Flex Your Super Powers: When a Design Ops Team Scales to Power CX
2021 • DesignOps Summit 2021
Gold

More Videos

Laine Riley Prokay

"Adding inclusivity to the career ladder took three years and this is still a living document with room for change."

Laine Riley Prokay

How DesignOps can Drive Inclusive Career Ladders for All

September 30, 2021

Victor Udoewa

"If diversity alone made research inclusive, then police could never be anti-black because they include black officers."

Victor Udoewa

Beyond Methods and Diversity: The Roots of Inclusion

March 26, 2024

Husani Oakley

"With great power comes great responsibility — that's our foundational context in exploring AI and ML."

Husani Oakley

Theme Three Intro

June 6, 2023

Jess Greco

"Changing outcomes means changing incentives on all fronts to shift the culture effectively."

Jess Greco

Creating a Basis for Change: Scaling Design Maturity

June 8, 2022

Shipra Kayan

"AI clustering gives a first stab at themes, but you have to move things around yourself."

Shipra Kayan

Make your research synthesis speedy and more collaborative using a canvas

January 24, 2025

Prayag Narula

"The focus is on helping teams take better decisions, not handholding them through every step."

Prayag Narula Abhinav Krishna

Dialing for Research: How to Reach the Unreachable

March 10, 2022

Sam Proulx

"The most important thing you can do is listen and watch and put assumptions aside about what is easy or hard."

Sam Proulx

Understanding Screen Readers on Mobile: How And Why to Learn from Native Users

June 6, 2023

Amy Evans

"Change is messy and it can be uncomfortable, much like baking bread—it’s hard to imagine sticky dough turning into a perfect loaf."

Amy Evans

How to Create Change

September 25, 2024

Greg Petroff

"In enterprise, we’re moving from systems of record, to systems of engagement, to systems of assets where context drives experience."

Greg Petroff

Everything is About to Change: Software as Material

June 8, 2016