Building the Rosenbot
Summary
- What does strategy look like in an AI world?
- What does eval-first mean?
- What the hell is going on deep inside an LLM?
- And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"I tried to do everything all by myself at first and it didn’t catch on because no one else knew I was doing it."
Shipra KayanHow we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up
September 30, 2021
"Warmth is a connecting principle that emerges from transcending self and attending to the world outside our own skin."
Daniel GloydTranscendent Experience (ZX) Design: A Design Framework for Human Connection
August 6, 2026
"There are as many versions of the website as stars in the sky because of language, currency, and device variations."
Erin WeigelUX Lessons from running more than 1,200 A/B Tests
July 10, 2024
"Political parties at the provincial level have their own mandates, which complicates unified service design."
Onur Kocan Ayhan EnsiciUnderstanding the Strategy for Civic Design in a Complex City: Istanbul
November 16, 2022
"If we just decide to ignore demographics we're missing huge swaths of the participant experience and an opportunity to better serve our users."
Megan CamposWhat Did I Miss? The Hidden Costs of Deprioritizing Diversity in User Research
March 12, 2021
"The bridge was made between math and physics in a way that hadn’t been done before."
Jennifer FraserWhat would Emmy Noether Do? Math, Models and Mulling in UX Research
March 29, 2023
"If you don’t have the time and organizational support, don’t do it because you only set a precedent of failure."
Sofia QuinteroBeyond Tools: The Messy Business of Implementing Research Repositories
March 10, 2022
"Steve invites us to look back at the history of our field so we can keep being difficult and courageous together."
Jemma AhmedTheme 2 Intro
March 26, 2024
"Telling your team's story goes beyond the walls. It's also about understanding your place within the team."
Jennifer KanyamibwaCreating the Blueprint: Growing and Building Design Teams
November 8, 2018
