Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Or choose a question:
More Videos
"You get a lot of different research outputs in a matter of minutes from your scope input."
Joerg Beringer Thomas GeisScaling User Research with AI: Continuous Discovery of User Needs in Minutes
June 10, 2025
"Stakeholders often fear participatory research because they don’t know the resources required or the outcomes they’ll get."
Nidhi Singh Rathore Amber DavisEmbracing participation to unlock deeper truths in commercial research
March 12, 2025
"Agile can feel like a cult, but the data helps show it’s working and not just a belief system."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"Google Maps’ Leaf visualization helps people normalize efficient driving, demonstrating the power of design in behavior change."
Jamie Beck Alexander Nina Gregg Shawn Petersen Bill DeRoucheyHow can you find your role in climate?
January 17, 2024
"We saw one insurance form took 14 minutes to fill, and that was longer than competitor journeys, helping identify a clear target for improvement."
Caroline VizeThe State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022
March 9, 2022
"Think of a creative professional as an ideas athlete or a creativity Olympian."
Dr. Karl JeffriesThe Science of Creativity for DesignOps
January 8, 2024
"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."
Alberto FerreiraMaking it Count: Developing a custom digital metric framework that works
October 15, 2021
"AI can act like a super eager intern who's very confident but sometimes gets things wrong."
Shipra Kayan Robert Kortenoeven Eileen TangEmerging principles for using AI in Design: What the product design team at Miro has learned from deeply integrating AI in their workflow
June 11, 2025
"Benchmarking has nearly doubled in two years, showing growing maturity in UX measurement across industries."
Dana Bishop2022: The Year UX Demonstrates its Business Impact
March 11, 2022
