Integrating generative AI into enterprise products: A case study from dscout
Summary
Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.
Key Insights
-
•
Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.
-
•
Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.
-
•
Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.
-
•
Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.
-
•
Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.
-
•
Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.
-
•
The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.
-
•
Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.
-
•
Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.
-
•
Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.
Notable Quotes
"Building on AI means giving up control, which is tough but necessary."
"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."
"The product is no longer fixed; after the first interaction, it diverges user by user."
"We’re not studying products anymore; we’re studying experiences in ecosystems."
"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."
"Users are building mental models of how to use AI while they’re using it."
"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."
"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."
"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."
"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."
Or choose a question:
More Videos
"Mastery means information is a strategic asset with quantifiable equity central to business identity and operations."
Bram WesselEnterprise Information Architecture
January 10, 2020
"Not every head of design is the right leader for every phase of organizational maturity."
John DevanneyThe Design Management Office
November 6, 2017
"We try to figure out who needs to be in the room to hear it — research is how we build team cohesion."
Rich MironovHow Can Product Managers and UXers Help Each Other (and Why are Product Folks so Annoying Sometimes)?
December 6, 2022
"The curse of hyper-focus means you lose peripheral vision and miss the larger context and customer needs."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"Recycling is not always intentional; often insights emerge accidentally when revisiting old data with new perspectives."
Iulia CornigeanuQuantQual Book Club: Small Data
March 8, 2024
"Write grammatically simple sentences with fewer commas and use the active voice to improve clarity."
Bruce GillespieLearning from journalism: Balancing impactful communication with compassionate storytelling
March 13, 2025
"Thirty to fifty percent of total efforts are typically spent on rework, which saps morale and wastes resources."
Rima Campbell Amrit S BhachuIncrease Productivity and Drive Business Impact
September 24, 2024
"Use custom systems for experts to quickly review and rate outputs, making feedback cycles much faster."
Peter Van DijckHands-on AI #2: Understanding evals: LLM as a Judge
October 15, 2025
"Reflexivity lets us examine our beliefs and practices during research to understand how we show up."
Bilan HashiThe Tension Between Story Collecting and Story Telling in Research
March 10, 2021
