Disrupting generative AI products with just-in-time consumer insights
Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Or choose a question:
More Videos
"Awareness leads to feeling stuck because we don’t know what to do or don’t feel empowered to act."
Victor UdoewaBeyond Methods and Diversity: The Roots of Inclusion
March 26, 2024
"I started thinking of the UI layer as the experimentation layer."
Bill ScottLean Engineering: Engineering for Learning and Experimentation in the Enterprise
May 14, 2015
"The context of use framework helps teams understand that different experiences aren't bad, just different."
Lija HoganContexts of Use: A Framework for Connection
December 9, 2021
"Traditional devops tools are more task and issue-centric, but designers think in processes and workflows."
Aurobinda Pradhan Shashank DeshpandeIntroduction to Collaborative DesignOps using Cubyts
September 9, 2022
"There's no such thing as over-communication when you're distributed; silence breeds assumptions."
Jilanna WilsonDistributed DesignOps Management
February 26, 2019
"Think of the context window like an intern's briefing document: would the intern be able to answer the question with this information?"
Peter Van DijckBuilding impactful AI products for design and product leaders, Part 3: Understand AI architectures: RAG, Agents, Oh My!
July 30, 2025
"There’s a lot of potential in UX research to go beyond product and influence strategy and C-suite decision making."
Nicole Aleong Michaela Mora Prayag Narula Brianna SylverWhat UX research can learn from other research practices [Advancing Research Community Workshop Series]
September 14, 2023
"The gap between what the output says and what the intention was is what we're going to call the reasoning gap."
Beth ChappellLeading through ambiguity: Supporting a design team relearning their craft
June 9, 2026
"It’s been a really challenging year for the design profession with layoffs continuing in 2025."
Llewyn PaineDay 1 Using AI in UX with Impact
June 10, 2025
