Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"Scaffolding is about hacking and doing things differently, not lowering the bar but delivering value continuously."
Ben Reason Aline Horta Majid Iqbal Fabiano LeoniMaking the system visible: The fastest path to better decisions
November 20, 2025
"Maybe you should consider making mobile the place you start your accessibility efforts."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
March 10, 2022
"It’s not all sunshine and rainbows; being new and working in a fast-moving startup means our process needs to stay flexible and adaptable."
Anna Nguyen Emily BroganWhy Our Voice of the Customer is Better Than Yours
March 10, 2022
"Eventually, I expect AI to help create designs by combining patterns from meta design systems and research, making design more about crafting experiences."
Abby Covert Tomer SharonPanel: Collaboration Tools
November 6, 2017
"Fix It Fridays are a dedicated time for the team to fix accessibility issues and make steady progress."
Phil HeskethDesigning Accessible Research Workflows
September 29, 2021
"You never get all your usability testers to respond, you always need to recruit more than you think."
Irina Tikhonova Kari DietrichSmall Wins, Big Impact: Leveraging and Elevating User Engagement
December 9, 2021
"Finding others within your organization who share your sustainability passion is the way to build momentum."
Tim FrickThe journey of building a sustainable design practice
April 23, 2025
"We could impact the end-to-end customer service and customer experience through our most underutilized asset: our store employees."
Catherine DubutBridging Physical and Digital Spaces: Approaches to Retail Service Design
March 18, 2021
"We recruited locally and from historically Black colleges and universities to build a diverse and talented innovation team."
Justin Entzminger Terrance Smith Tracy M. Colunga Mai-Ling GarciaRisk and Reward: How to Diversify the Field of Civic Innovators and Designers
November 17, 2022
