Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"Designing for edge cases can lead to solutions everyone wants to use because they’re more comfortable and inclusive."
Billy CarlsonIdeation tips for Product Managers
December 6, 2022
"Both iOS and Android have built-in screen magnification and voice control that don’t require extra software."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
November 17, 2022
"When systems fail, Jigad works best within a framework — innovation that must scale happens with an ecosystem view."
Dan WillisEnterprise Storytelling Sessions
May 13, 2015
"An org chart is about formal reporting; a relationship map is about networking and collaboration across silos."
Michael PolivkaScaling Design through Relationship Maps
November 7, 2017
"Different AI models have wildly different views of the same content; the model choice makes a big difference."
Karen McGrane Jeff EatonAI for Information Architects: Are the robots coming for our jobs?
November 21, 2024
"One researcher per team for at least three days a week was considered an outrageous luxury at GDS in 2013."
Leisa ReicheltOpening Keynote: Operating in Context
November 7, 2018
"Trust takes time, hard work, and consistency to build, especially in times of change when time is a luxury we don't have."
Kim Holt Emma Wylds Pearl Koppenhaver Maisee XiongA Salesforce Panel Discussion on Values-Driven DesignOps
September 8, 2022
"Listening with tactical empathy means understanding the full emotional journey during times of change."
Jacqui Frey Alison RandSetting the Table for Dynamic Change
October 24, 2019
"If you think about things this way, that’s when you actually become the CEO of the experience."
How to Identify and Increase your "Experience Quotient"
June 15, 2018
Latest Books All books
Dig deeper with the Rosenbot
What practical steps did Flight Center take to scale research participation without sacrificing insight quality?
How can program or project managers support design ops activities when no dedicated ops role exists?
What are the key financial metrics that influence product and service design decisions?