Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

When AI Becomes the User’s Point Person—and Point of Failure

Thursday, August 7, 2025 • Rosenfeld Community
Share the love for this talk
When AI Becomes the User’s Point Person—and Point of Failure
Speakers: Heidi Trost
Link:

Summary

Imagine slipping on a sleek pair of smart glasses. Not only do you look sharp, the glasses capture everything you see, hear, and do. Your AI assistant—built into the glasses and synced to your email, social media accounts, health apps, and finances—manages your life. It’s tasked with paying bills, booking trips, replying to messages, even helping you swipe right. Over time, you find yourself chitchatting with your AI assistant. You call him Charlie. Now imagine you’re a threat actor. That trust between user and AI assistant? It’s your entry point. If your product is powered by AI, you’re not just designing features—you’re designing an entire relationship. You’re designing Charlie. Let’s talk about where that goes wrong—and how to get it right.

Key Insights

  • Users often do not understand why AI-powered systems request extensive personal data, increasing privacy risks.

  • Trust in AI agents can become excessive, creating new vectors for manipulation by threat actors.

  • Security issues typically occur beneath the surface until alerts disrupt the user experience, often causing frustration.

  • Prompt injection attacks pose a novel threat where malicious inputs manipulate AI agents to access sensitive user data.

  • Multimodal AI interfaces introduce complexity in security decisions, increasing chances for user errors.

  • Secure by default settings reduce burden on users and improve overall protection without requiring user intervention.

  • Cross-disciplinary collaboration between UX, security, product, legal, and compliance teams is crucial for safer AI design.

  • Users need clear, contextual guidance during onboarding to make informed decisions about data sharing and security settings.

  • Transparency about AI limitations and giving users the option to reverse AI actions are essential for building trust.

  • Threat actors are likely to exploit growing AI access to personal data and automate vulnerabilities discovery.

Notable Quotes

"When a product is powered by AI, you're not just designing the features; you are designing an entire relationship."

"Charlie is like the most annoying coworker who constantly surfaces problems but never offers solutions to Alice."

"Threat actors probably know your system better than you do and are looking for any entry points to exploit."

"Alice often perceives Charlie as just another barrage of alerts filled with jargon she doesn't understand."

"Prompt injection attacks can trick AI agents into accessing private data like emails without the user realizing."

"People become incrementally more comfortable giving away data because they see the value AI provides."

"We need secure defaults that protect users out of the box without them having to figure it out."

"Alert fatigue is real; users can't be burdened with constant security decisions or they'll ignore them."

"Giving users the ability to reverse AI-driven actions is critical but currently underexplored."

"If Charlie has been tampered with, Alice needs a clear way to be alerted that she shouldn't trust it."

Ask the Rosenbot
Jonathan Fairman
Integrating generative AI into enterprise products: A case study from dscout
2024 • Designing with AI 2024
Gold
Jess Greco
Creating a Basis for Change: Scaling Design Maturity
2022 • Design at Scale 2022
Gold
Michaela Mora
Advanced Concept Testing Approaches To Guide Product Development and Business Decisions
2022 • Advancing Research 2022
Gold
Marjorie Stainback
What Research Ops Professionals Have Learned from COVID-19
2020 • Advancing Research Community
Megan Blocker
Panel: Excellence in Impact
2024 • Advancing Research 2024
Gold
Alexandra Schmidt
Enterprise UX Playbook
2022 • Enterprise Community
Smitha Papolu
Theme 3 Discussion
2019 • Enterprise Experience 2019
Gold
Scott Plewes
Why Isn't Your UX Approach Going Viral?: A Mathematical Model
2023 • Advancing Research 2023
Gold
Iram Shah
Closing Keynote: The View from the Top
2019 • Enterprise Experience 2019
Gold
Marc Fonteijn
Increase your confidence, influence, and impact (through a Professional Community)
2024 • Advancing Service Design 2024
Gold
Joseph Williams
Unlocking impact and influence through inclusive hiring in research
2021 • Advancing Research Community
Peter Van Dijck
Building the Rosenbot
2024 • Designing with AI 2024
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • Civic Design 2021
Gold
Jon Fukuda
Design Planning and Management Support
2023 • DesignOps Summit 2023
Gold
Andrew Custage
The Digital Journey: Research on Consumer Frustration and Loyalty
2023 • Advancing Research 2023
Gold
Jim Kalbach
Jazz Improvisation as a Model for Team Collaboration
2019 • Enterprise Experience 2019
Gold

More Videos

Dan Willis

"Anytime you have to justify something by saying just this one time, you know you’re going down a bad road."

Dan Willis

Enterprise Storytelling Sessions

May 13, 2015

Sarah Flamion

"Marketers don’t try very interesting things when their confidence in data is low, which can create a downward spiraling reinforcing loop."

Sarah Flamion

Complex Problem? Add Clarity by Combining Research and Systems Thinking

March 31, 2020

Frances Yllana

"It’s really hard when all that tacit knowledge walks out the door every time someone leaves."

Frances Yllana Jorge Arango Maria Taylor Briana Thomas

The Big Question about Impact: A Panel Discussion

September 24, 2024

Louis Rosenfeld

"As a starter, I’m great at beginning things, but the company has the infrastructure to maintain them."

Louis Rosenfeld

The Rosenbot and the Rosenverse: An AMA with Lou Rosenfeld

June 5, 2024

Sara Conklin

"Don’t cold reach out to people with ‘let’s connect’; instead, ask very specific questions about topics they care about."

Sara Conklin

A UXer’s 12-Month Journey from Climate Concern to Climate Credibility

June 26, 2025

Susan Simon-Daniels

"The fundamental things do apply as time goes by."

Susan Simon-Daniels

War Stories LIVE! Susan Simon-Daniels

March 30, 2020

Deanna Zandt

"I finally started saying, I understand your budget, but I have one too, and I deserve to be paid."

Deanna Zandt

The Unspoken Complexity of “Self-Care” with Deanna Zandt

July 21, 2022

Kara Kane

"Civic design and government design practice have reached many milestones in the past 10 years."

Kara Kane

Theme One Intro

November 16, 2022

Jayne Engle

"We are all multitudes of identities and experiences, far beyond what any screen can show."

Jayne Engle Tanya Chung-Tiam-Fook

Civic Design for the Next Seven Generations—A Discussion on Sacred Civics

August 25, 2022