Integrating generative AI into enterprise products: A case study from dscout
Summary
Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.
Key Insights
-
•
Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.
-
•
Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.
-
•
Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.
-
•
Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.
-
•
Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.
-
•
Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.
-
•
The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.
-
•
Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.
-
•
Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.
-
•
Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.
Notable Quotes
"Building on AI means giving up control, which is tough but necessary."
"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."
"The product is no longer fixed; after the first interaction, it diverges user by user."
"We’re not studying products anymore; we’re studying experiences in ecosystems."
"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."
"Users are building mental models of how to use AI while they’re using it."
"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."
"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."
"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."
"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."
Or choose a question:
More Videos
"Building trust in the user is fundamental, especially in digital products like our doggy daycare app example."
Yunyan Li Anna Le Jen KimUX Best Practices
June 11, 2021
"We went from stitching wireframes to actually designing in code—no more simulacra."
Allan LowsonRehashing the Double Diamond: Collaborating across functions with AI-assisted prototyping
June 9, 2026
"Racism doesn’t just hurt people like me; you’re hurting yourself by perpetuating beliefs that deny a greater humanity."
Denise Jacobs Nancy Douyon Renee Reid Lisa WelchmanInteractive Keynote: Social Change by Design
January 8, 2024
"The sum is greater than the parts—how qualitative and quantitative research play together."
Renee BouwensLanding Product Impact: Aligning Research as a Foundational Driver for Delivering the World’s Best Products
December 15, 2023
"Democratization is not research anarchy where anybody can talk to any customer for any reason with no oversight."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"This talk is more than anything else, it’s about confidence."
Sam LadnerHow Research Can Drive Strategic Foresight
March 9, 2022
"Designers adhere to standards not scientific ones, but design rigor tailored to dealing with complexity."
Yoel SumitroActions and Reflections: Bridging the Skills Gap among Researchers
March 9, 2022
"Helping teams is much more like managing a children’s party than engineering a bridge."
Dave HoraResearch in the Face of Complexity: New Sensibility for New Situations
August 27, 2025
"Visual design quality actually affects perceived trustworthiness."
Daniel J. RosenbergDesigning with and for Artificial Intelligence
August 11, 2022