Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Dig deeper—ask the Rosenbot:
More Videos
"We all walked away better for it."
Randolph Duke IIWar Stories LIVE! Randy Duke II
March 30, 2020
"This is a moment to rethink identity beyond UX and get creative with income streams and career paths."
Corey Nelson Amy SanteeLayoffs
November 15, 2022
"Every board member can interpret an NPS score differently depending on their role and responsibilities."
Landon BarnesAre My Research Findings Actually Meaningful?
March 10, 2022
"There was a founder who said our job was to coach people to not need us anymore, which is the kind of mindset I wish was more common."
Amy BucherHarnessing behavioral science to uncover deeper truths
March 12, 2025
"With QFI, we go upstream: simulate, model, predict user behavior before shipping, not just react after."
David SternbergUncovering the hidden forces shaping user behavior
July 17, 2025
"Assessing feedback urgency means understanding if the issue blocks critical work or if there are workarounds."
Deanna SmithLeading Change with Confidence: Strategies for Optimizing Your Process
September 23, 2024
"Work on yourself. Understand your identity and privilege before trying to serve others."
Jennifer StricklandAdopting a "Design By" Method
December 9, 2021
"My path into design Ops was winding—I started as a professional ballet dancer before falling in love with academia and design."
Rachel Posman John CalhounA Closer Look at Team Ops and Product Ops (Two Sides of the DesignOps Coin)
November 19, 2020
"Grandmas provide grounding, resilience, and long-term perspective often through rituals and storytelling."
Gina MendoliaTherapists, Coaches, and Grandmas: Techniques for Service Design in Complex Systems
December 3, 2024
Latest Books All books
Dig deeper with the Rosenbot
How does the system classify different user roles and stakeholders from input data?
What strategies did Sean and Sarah use to create a shared language and structure for managing government design knowledge?
How can design ops build onboarding processes that reduce new hire confusion and fragmentation?