Summary
A 30-minute deep-dive into the building of the Rosenbot. We’ll get both hands-on practical and likely a bit philosophical. What does it take to build a useful AI assistant? What does it mean for a business, strategically? And how do we make sure we are building the future that we want, while doing all this? Take-aways: What does strategy look like in an AI world? What does eval-first mean? What the hell is going on deep inside an LLM? And what does all that mean for the future we want to build?
Key Insights
-
•
Generative AI is a new design material, similar to previous technological advancements.
-
•
The Rose Bot was created to help users navigate Rosenfeld Media's extensive resources.
-
•
Designers need to rethink their processes and techniques to accommodate generative AI systems.
-
•
Evaluation of AI outputs must involve both expert and end-user perspectives to ensure reliability and usefulness.
-
•
Building generative AI systems requires the development of specialized new tools and techniques for prompt engineering and evaluation.
-
•
Conversations with AI are not simple and require structured conversation logic to improve user experience.
-
•
A significant focus on research is necessary to adapt existing methodologies for testing and evaluation in the context of generative AI.
-
•
The role of UX designers is evolving; they must embrace the challenge of co-creating alongside AI technologies.
-
•
The engineering mindset shouldn't dominate; user involvement is crucial in shaping AI technology's practical applications.
-
•
New methods in user research should reflect the distinct characteristics of generative AI interactions.
Notable Quotes
"I was not a fan of crypto and VR, but when I saw my kids using GPT, the use case became immediately clear."
"This is a new design material; we need to reinvent how we design."
"The job market is crazy; junior UX people are questioning if they joined the right discipline."
"If we build a chatbot, it needs to read everything and analyze it for semantic meaning."
"If you don’t do eval, you’re just building a demo — demos are easy, good quality production is hard."
"Spending 30 minutes on one prompt in a usability test is super fascinating."
"The future is co-created by users and technology; it's not solely driven by tech."
"We need to research this stuff, but we must make up how we research it."
"Let's invent tools that are good for people; it's equally hard to build bad products as it is to build good ones."
"Research right now is super exciting; how do we know if generative AI outputs are good?"
















More Videos

"We help organizations build relationships with their audiences just like people do."
Noz UrbinaRapid AI-powered UX (RAUX): A framework for empowering human designers
May 1, 2025

"What are our rules of engagement for design?"
Jim KalbachJazz Improvisation as a Model for Team Collaboration
June 4, 2019

"What would you do if info sets said you can't use this tool?"
Holly ColeUnderstanding Experiences: When you have to do more than work
November 8, 2018

"Hello, everyone, my name is Jack Behar and I will be leading this session."
Jack BeharHow to Build Prototypes that Behave like an End-Product
December 6, 2022

"You don't want to miss out on one of the really cool parts about the conference, which is going home with a bunch of cool swag."
Bria AlexanderOpening Remarks
October 3, 2023

"We had to look outside and ask if we were ready for AI, as the world was moving on without us."
Erika FlowersAI-Readiness: Preparing NASA for a Data-Driven, Agile Future
June 10, 2025

"Teachers need actionable insights, not just raw data."
Kristin SkinnerFive Years of DesignOps
September 29, 2021

"How can a group of people come together and spontaneously create something that is legendary?"
Jim KalbachJazz Improvisation as a Model for Team Collaboration
November 6, 2017

"The future solutions will come from harnessing creativity at scale."
Surya VankaUnleashing Swarm Creativity to Solve Enterprise Challenges
June 10, 2021