Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Building the Rosenbot
Gold
Tuesday, June 4, 2024 • Designing with AI 2024
Share the love for this talk
Building the Rosenbot
Speakers: Peter Van Dijck
Link:

Summary

A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?

Key Insights

  • Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.

  • Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.

  • Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.

  • Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.

  • Effective evaluation requires combining human expert and end-user feedback alongside automated tools.

  • A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.

  • ‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.

  • New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.

  • UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.

  • Co-creation between users and technology defines how AI applications evolve and succeed or fail.

Notable Quotes

"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."

"This AI stuff is a new design material, just like the internet or mobile was before."

"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."

"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."

"There’s a lot of steps behind the scenes to make conversations useful and safe."

"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."

"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."

"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."

"Co-creation between users and technology determines if this AI evolves to be useful or harmful."

"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."

Ask the Rosenbot
Robert Schwartz
We're Here for the Humans
2017 • Enterprise Experience 2017
Gold
Peter Van Dijck
Building impactful AI products for design and product leaders, Part 1: The new product journey
2025 • Rosenfeld Community
Patrick Commarford
Design Staffing for Impact
2024 • DesignOps Summit 2020
Gold
Andrea Gallagher
The Problem Space
2019 • Advancing Research Community
Tatyana Mamut
Opening Keynote: Breaking Conway's Law--or How to Work Differently and Not Ship Your Org Chart
2019 • Enterprise Experience 2019
Gold
Cennydd Bowles
Exit Interview #2: Rediscovering the ethical heart of design
2025 • Rosenfeld Community
Heidi Trost
To Protect People, You Have to Protect Information: A Human-Centered Design Approach to Cybersecurity
2025 • Rosenfeld Community
Stephanie Wade
Building and Sustaining Design in Government
2021 • Civic Design 2021
Gold
Ted Booth
Discussion
2016 • Enterprise UX 2016
Gold
Frank Duran
Partnership Playbook: Lessons Learned in Effective Partnership
2024 • Enterprise Experience 2020
Gold
Dane DeSutter
Keeping the Body in Mind: What Gestures and Embodied Actions Tell You That Users May Not
2024 • Advancing Research 2024
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Megan Nipe
Human-Centered Design for Engagement: Maturing from Newsletterville to Personalized, One-to-One Messaging
2021 • Civic Design 2021
Gold
Louis Rosenfeld
Coffee with Lou
2024 • Rosenfeld Community
Louis Rosenfeld
Opening Remarks
2023 • Design in Product 2023
Gold
Sylvie Abookire
A Civic Designer's Guide to Mindful Conflict Navigation
2022 • Civic Design 2022
Gold

More Videos

Ned Dwyer

"Research ops was actually there before research at TravelPerk, supporting designers first."

Ned Dwyer Emily Stewart James Wallis

The Intersection of Design and ResearchOps

September 24, 2024

Michael Land

"There are lots of little mushroom patches of design sprouting up, but they're disconnected and not sharing."

Michael Land

Establishing Design Operations in Government

February 18, 2021

Sam Proulx

"The first organization to solve accessibility for new modalities like AR/VR will gain a massive, lasting advantage."

Sam Proulx

To Boldly Go: The New Frontiers of Accessibility

November 18, 2022

Vasileios Xanthopoulos

"We set the system usability scale score as a KPI for all products to improve measurement and usability."

Vasileios Xanthopoulos

A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale

January 8, 2024

Andrew Custage

"Sometimes vocal feedback reveals problems that digital behavior can’t see—like offline factors affecting perception."

Andrew Custage Michael Mallett

The Digital Journey: Research on Consumer Frustration and Loyalty

March 29, 2023

Marjorie Stainback

"Our program changed trains people to create high quality, unbiased user research."

Marjorie Stainback Kelsey Kingman

Transforming Strategic Research Capacity through Democratization

October 24, 2019

Alla Weinberg

"75% of cross-functional teams are dysfunctional because silos perpetuate themselves, even in cross-functional teams."

Alla Weinberg

Cross-Functional Relationship Design

December 6, 2022

Jerome “Axle” Brown

"Change is the only constant, so designing insight sharing with self-directed learning is critical."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Jen Crim

"Our offices have stand-up desks, nice collaboration areas, and comfy seating with a fresh, on-brand look."

Jen Crim Jess Quittner Saritha Kattekola Alex Karr Gurbani Pahwa

Culture, DIBS & Recruiting

June 11, 2021