Building the Rosenbot
Summary
- What does strategy look like in an AI world?
- What does eval-first mean?
- What the hell is going on deep inside an LLM?
- And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material requiring fresh approaches distinct from prior UX methods.
-
•
Rose Bot leverages retrieval augmented generation to semantically search Rosenfeld’s extensive UX content.
-
•
Conversation logic beyond the LLM is essential to route, classify intents, and ensure safe responses.
-
•
Evaluation (eval) of AI is fundamentally challenging due to its non-deterministic input and output.
-
•
Effective evaluation requires combining human expert and end-user feedback alongside automated tools.
-
•
A third of a project’s budget and time should be dedicated solely to rigorous AI evaluation.
-
•
‘Prompt deep dive’ usability testing spends extended time on individual prompts to deeply understand interactions.
-
•
New tooling is emerging specifically for tracing conversations, prompt engineering, and observability in AI.
-
•
UX roles remain vital in AI development by inventing new research techniques and ensuring user-centered design.
-
•
Co-creation between users and technology defines how AI applications evolve and succeed or fail.
Notable Quotes
"When GPT came out, my kids immediately adopted it at school and couldn’t pry it from their dead hands."
"This AI stuff is a new design material, just like the internet or mobile was before."
"The Rose Bot has read everything—every piece of Rosenfeld’s knowledge—to help users access it."
"Building generative AI tools is not scary, it’s just very different from the last 20 years of building products."
"There’s a lot of steps behind the scenes to make conversations useful and safe."
"Eval is the engineering word for evaluation, but also what researchers and UX designers naturally do."
"Without proper evaluation, you’re just building a demo – demos are easy, quality production is hard."
"We invented the ‘prompt deep dive’ technique to spend lots of time on one prompt to deeply understand it."
"Co-creation between users and technology determines if this AI evolves to be useful or harmful."
"We should feel okay to throw away old assumptions and tooling and invent new techniques for this new world."
Or choose a question:
More Videos
"The ultimate goal in conflict is to engage with the tension while reducing the friction between humans."
Laura WeissTurn Down the Heat: 3 Ways to Handle Conflict in the Moment
November 20, 2024
"To have a strategy you need to think about how to prioritize, because you can’t do all the research requests that come in."
George Zhang Molly StevensUX Research Excellence Framework
March 11, 2021
"The Shakers’ principled approach to design was a precursor to Bauhaus’s form follows function and today’s user-centered values."
Daniel GloydWarming the User Experience: Lessons from America's first and most radical human-centered designers
May 9, 2024
"Sometimes it's just not even related to the research skills, sometimes relationships really matter."
Megan Blocker Mujtaba Hameed Victor UdoewaPanel: Excellence in Impact
March 25, 2024
"Make an uncredible experience credible. That was the directive from organizational resistance to UX."
Dan WillisEnterprise Storytelling Sessions
June 3, 2019
"Sometimes hidden blockers come from power structures that prefer things stay the way they are."
Dave HoraAdvice for Establishing Research
December 8, 2022
"You cannot do design today, even product or service design, without being systemically engaged."
Paul Pangaro, PhDSystems Disciplines: Table Stakes for 21st Century Organizations
June 6, 2023
"Low UX maturity is often a symptom of a bigger problem in how you build product, not just design adoption."
Bob Baxley Sara Asche Anderson Sharon Bautista Frank Duran Jamie Kaspszak Abbey Smalley Sylas SouzaTheme 4: Discussion
January 8, 2024
"Data analysts and user researchers rarely work together effectively, which is one of the biggest missed opportunities."
Andrew MichaelBuilding a Product Insights Team
March 10, 2022
Latest Books All books
Dig deeper with the Rosenbot
Why do current capitalist incentives make systemic and community-focused tech solutions difficult to implement?
What role does documenting accessibility trade-offs play in future-proofing design systems for AI?
How can UX teams use the exercise of banning jargon to improve collaboration and shared understanding?