Summary
Join Katie Johnson, Head of Consumer Insights at PanasonicWELL, to explore the future of developing products and services atop Generative AI technology. This talk will cover how to think about experimenting with users today for use cases that don’t yet exist at scale, and how to bring insightful findings to leadership to make strategic decisions quickly. Katie will shed light on methods and strategies she’s employed in building products with cutting-edge technology throughout her career in agency life, blockchain, Google’s 0 to 1 environment in Assistant, and now at PanasonicWELL.
Key Insights
-
•
Building AI products requires giving up control over user experience as outputs vary uniquely per user.
-
•
Traditional usability testing is insufficient for AI products; longitudinal testing with repeated sessions is critical.
-
•
Human-in-the-loop 'Wizard of Oz' testing effectively simulates AI to rapidly test assumptions before building.
-
•
Model drift in LLMs can cause unpredictable or inappropriate responses, requiring mechanisms like turn capping.
-
•
Users form evolving relationships with AI assistants, which need careful design and testing over time.
-
•
Many product orthodoxies based on fixed experiences expire with generative AI and must be reinvented.
-
•
Failure early and often is essential in AI product development due to its inherent unpredictability.
-
•
Users may lack mental models for AI assistants, making onboarding and user education a bigger challenge.
-
•
Asynchronous, multi-threaded conversations are natural for humans but difficult for bots to handle accurately.
-
•
AI can automate synthesis tasks, enabling humans to focus on creativity, accountability, and authenticity.
Notable Quotes
"Building on AI and generative AI means giving up control."
"If the folks that design this tech don’t know how it works, it’s okay if you don’t know either."
"Alignment inevitably means changing minds."
"Everyone’s gonna be wrong. If you’re right all the time, you’re probably not taking big enough swings."
"With LLMs, the product itself is no longer a dependent variable; experiences diverge per user."
"We are studying experiences in ecosystems, not just products anymore."
"Failing early is critical because you can’t control the model."
"The promise of an LLM assistant requires users to build new mental models while using it."
"Turn capping helps reset the model to prevent drift but introduces new problems around memory."
"Creativity, accountability, and authenticity are going to be the new markers of humanity."
Dig deeper—ask the Rosenbot:
More Videos
"Researchers were obsessed with scientific rigor while designers used balance, feelings, or intuition as their rigor."
Yoel SumitroActions and Reflections: Bridging the Skills Gap among Researchers
March 9, 2022
"We as a society are taught to care about people, rarely taught to care about ourselves."
Zariah CameronReDesigning Wellbeing for Equitable Care in the Workplace
September 23, 2024
"Forrester found that design thinking teams move twice as fast from idea to market because they’re closer to their users."
Doug PowellDesignOps and the Next Frontier: Leading Through Unpredictable Change
September 11, 2025
"It takes at least one member of a team to choose to set healthy standards for collaboration."
Darian DavisLessons from a Toxic Work Relationship
January 8, 2024
"New ideas and changes to business trajectories rank pretty far down at only 18% of business leaders seeing it as a value of design."
Mike OrenWhy Pharmaceutical's Research Model Should Replace Design Thinking
March 28, 2023
"Design Ops brings the joy back into design."
Jacqui FreyScale is Social Work
March 19, 2020
"The V2MOM stands for Vision, Values, Methods, Obstacles, and Measures and is a framework every Salesforce employee uses."
Kim Holt Emma Wylds Pearl Koppenhaver Maisee XiongA Salesforce Panel Discussion on Values-Driven DesignOps
September 8, 2022
"Designing for the average means designing for nobody."
Samuel ProulxFrom Standards to Innovation: Why Inclusive Design Wins
September 10, 2025
"This is not a checklist or promotion tool; you don’t have to excel in everything to grow."
Laine Riley ProkayHow DesignOps can Drive Inclusive Career Ladders for All
September 30, 2021
