Building the Rosenbot
Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"If you get the interview, say, I don’t have experience in that field, but here’s how I solved similar complex problems."
Dave Hoffer Joanne WeaverUX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
May 21, 2025
"HITS indexes individual insights, not just whole reports, so you can drill down into the actual claims behind evidence."
Matt DuignanHITS, Microsoft's internal human insight system: From research library to living body of knowledge
July 16, 2019
"Feedback is data, not personal — once you view it that way, it feels less like criticism and more like improvement information."
Bria Alexander Laura Gatewood Corey Long Daniel Orbach Laine Prokay Deanna SmithThe Big Question about Resilience: A panel discussion
September 23, 2024
"Discovery doesn’t usually come with a launch party or even its own Jira ticket, but it’s what turns speed into progress."
Bianca JeffersonFrom Sprints to Systems: Operationalizing Continuous Discovery Through DesignOps
September 10, 2025
"Quantitative data boils complex situations down to single numbers that are easy to grasp and discuss."
Rima Campbell Amrit S BhachuIncrease Productivity and Drive Business Impact
September 24, 2024
"Inside every challenge is an opportunity, and letting your partners challenge your assumptions is necessary."
Melinda BelcherBridging the Gap: Making the Most of the Differences Between Agency and Enterprise
January 8, 2024
"You want to let participants know it’s a bit non-typical and they are not being tricked or evaluated."
Marc Majers Tony TurnerInterrupted UX - Add A Dose of Reality To Usability Testing
March 11, 2022
"Audacity means being willing to do things that seem dumb or risky because that’s how you innovate."
Deanna Washington Tim Allen Jeff Courcelle John Maeda Matt Raw Erica TjaderScaling Success: Paving the Path from DesignOps to VP
October 4, 2023
"It comes down to confidence and comfort to articulate why something has value and whether it’s working or not."
Vicky Teinaki Michele Marut Tim ParmeeShort Take #3: UX/Product Lessons from Your Industry Peers
December 6, 2022
