Disrupting generative AI products with just-in-time consumer insights
Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Or choose a question:
More Videos
"If something looks rough, people don’t treat it like a declaration. They treat it like a draft."
Joe ElmendorfShitty Maps: A Trojan Horse for Sensemaking
July 9, 2026
"Designers want more time to prototype and really ensure what ships is successful, but we just don’t have enough people or time."
Kevin M. HoffmanTheme 2: Enterprise Team Journey
June 3, 2019
"There's no universal design for all aging factors; customization is key."
Rittika BasuAge and Interfaces: Equipping Older Adults with Technological Tools
February 23, 2023
"Power is the ability to change another person’s reality."
George AyeThat Quiet Little Voice: When Design and Ethics Collide
November 16, 2022
"Leadership buy-in now means we get to focus on bigger, generative research questions instead of just tactical usability studies."
Andreas Huebner Amy Takata Craig BrookesWhat Is It Like To Be Part of The UX Team at Compass?
March 11, 2021
"Women are 70% more likely to walk or use public transit, combining trips that are much harder with snow than driving a car."
Kaitlin TaskerFast and Fearless Inclusive Research
March 27, 2023
"I’ve started giving up on product management and focusing more on product strategy and senior executives."
Steve Portigal Alba Villamil Sam LadnerThe Future of Research: Bridging the Gaps
July 29, 2021
"Invitational language allows participants to be in charge of a part of the interview, making the exchange authentic."
Bilan HashiThe Tension Between Story Collecting and Story Telling in Research
March 10, 2021
"Our job is to make sure the people making decisions actually listen."
Jemma AhmedTheme 1 Intro
March 10, 2026
