Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Integrating generative AI into enterprise products: A case study from dscout

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Integrating generative AI into enterprise products: A case study from dscout
Speakers: Jonathan Fairman and Kevin Johnson
Link:

Summary

Join dscout for an in-depth case study walking through a first-year journey of integrating Generative AI into an established enterprise product. In a session tailored specifically for product teams, the speakers will delve into the paradigm shifts in UX professionals to technical relationships, strategies, and processes. dscout’s Jonathan Fairman, VP of Product, and Kevin Johnson, Head of AI, will discuss how the stochastic nature of AI technologies led to the move of a piloting approach to development and traditional wireframe testing of concepts no longer possible.

Key Insights

  • Building AI products requires giving up control over the product's behavior due to the unpredictable nature of LLMs.

  • Traditional usability testing assumptions break down with AI because each user experience dynamically diverges after the first interaction.

  • Longitudinal testing is critical for AI products to understand how user relationships with models evolve over time.

  • Wizard of Oz testing (humans simulating AI) is a low-cost method to validate assumptions before building complex AI models.

  • Model drift in LLMs causes unpredictable and sometimes harmful behavior, necessitating turn-capping or memory features to maintain consistency.

  • Users build their mental models for AI assistants dynamically while using them, unlike fixed software products.

  • The stochastic nature of AI results means identical commands can produce different outcomes, affecting user mastery and trust.

  • Multi-dimensional prioritization frameworks, like importance vs. difficulty, help teams align on assumptions to test first in AI product development.

  • Recognizing failure as inevitable and building failure metrics accelerates learning and innovation in generative AI.

  • Humans will retain roles centered on creativity, critical thinking, accountability, and authenticity in a future transformed by AI.

Notable Quotes

"Building on AI means giving up control, which is tough but necessary."

"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."

"The product is no longer fixed; after the first interaction, it diverges user by user."

"We’re not studying products anymore; we’re studying experiences in ecosystems."

"Failure is inevitable—if you’re right all the time, you’re not taking big enough swings."

"Users are building mental models of how to use AI while they’re using it."

"Testing AI products requires longitudinal methods to see how relationships and experiences evolve."

"Wizard of Oz testing with humans playing bots can reveal insights without building a full model."

"Asynchronous conversations have nested threads humans track easily but are hard for bots to replicate."

"Creativity, accountability, and authenticity will be the new markers of humanity alongside AI."

Ask the Rosenbot
Bria Alexander
Opening Remarks
2022 • Civic Design 2022
Gold
Josh Clark
Sentient Design: Crafting Intelligent Interfaces with AI
2026 • Designing with AI 2026
Conference
Simon Wardley
Maps and Topographical Intelligence
2019 • Enterprise Community
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Rosenfeld Community
Liz Ebengo
The Burden on Children: The Cost of Insufficient Post-Conflict Services and Pathways Forward
2024 • Advancing Service Design 2024
Gold
Weidan Li
Qualitative synthesis with ChatGPT: Better or worse than human intelligence?
2024 • Designing with AI 2024
Gold
Theresa Neil
Designing for Wellness: Specializing in Healthcare
2024 • Rosenfeld Community
Bria Alexander
Opening Remarks
2021 • Design at Scale 2021
Gold
Jennifer Kong
Journeying toward AI-assisted documentation in healthcare
2024 • Designing with AI 2024
Gold
Jen Cardello
Standardizing Product Merits for Leaders, Designers, and Everyone
2018 • Enterprise Experience 2018
Gold
James Chudley
Decarbonising User Journeys: How minimising enables us to do more with less
2025 • Climate UX Interest Group
Jeff Gothelf
The Intersection of Lean and Design
2019 • Enterprise Community
Etienne Fang
The Power of Care: From Human-Centered Research to Humanity-Centered Leadership
2021 • Advancing Research 2021
Gold
Kaaren Hanson
Stop Talking, Start Doing
2017 • Enterprise Experience 2017
Gold
Lada Gorlenko
Theme 1: Discussion
2024 • Enterprise Experience 2020
Gold
Kristin Taylor
Building Bridges Across Organizational Silos
2022 • Civic Design 2022
Gold

More Videos

Yunyan Li

"Love principle is about emotionally supporting users, celebrating achievements, and creating visually beautiful products."

Yunyan Li Anna Le Jen Kim

UX Best Practices

June 11, 2021

Allan Lowson

"We introduced a context diamond and prototype diamond to impose guardrails around AI-assisted prototyping."

Allan Lowson

Rehashing the Double Diamond: Collaborating across functions with AI-assisted prototyping

June 9, 2026

Denise Jacobs

"Every day is a chance to check: is this how I want to be? This is a long-haul, marathon effort."

Denise Jacobs Nancy Douyon Renee Reid Lisa Welchman

Interactive Keynote: Social Change by Design

January 8, 2024

Renee Bouwens

"Analytics and research should work closely to explain what is happening and why it is."

Renee Bouwens

Landing Product Impact: Aligning Research as a Foundational Driver for Delivering the World’s Best Products

December 15, 2023

Jemma Ahmed

"Knowledge multiplies the more it is shared; enabling others makes your insights stronger, not your position weaker."

Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher Nash

Democratization: Working with it, not against it [Advancing Research Community Workshop Series]

July 24, 2024

Sam Ladner

"Rigour really is just a method; it’s knowing why you made the choices and tradeoffs you did."

Sam Ladner

How Research Can Drive Strategic Foresight

March 9, 2022

Yoel Sumitro

"Design research faces design complexity, which deals with the specific, intentional, and the non-existing."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Dave Hora

"Democratization of research is interesting but not a one-size-fits-all; it needs quick assessment of what a team truly needs."

Dave Hora

Research in the Face of Complexity: New Sensibility for New Situations

August 27, 2025

Daniel J. Rosenberg

"AI-generated UX is not about replacing designers, but removing grunt work to focus on higher-order design."

Daniel J. Rosenberg

Designing with and for Artificial Intelligence

August 11, 2022