Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

When AI Becomes the User’s Point Person—and Point of Failure
Thursday, August 7, 2025 • Rosenfeld Community
Share the love for this talk
When AI Becomes the User’s Point Person—and Point of Failure
Speakers: Heidi Trost
Link:

Summary

Imagine slipping on a sleek pair of smart glasses. Not only do you look sharp, the glasses capture everything you see, hear, and do. Your AI assistant—built into the glasses and synced to your email, social media accounts, health apps, and finances—manages your life. It’s tasked with paying bills, booking trips, replying to messages, even helping you swipe right. Over time, you find yourself chitchatting with your AI assistant. You call him Charlie. Now imagine you’re a threat actor. That trust between user and AI assistant? It’s your entry point. If your product is powered by AI, you’re not just designing features—you’re designing an entire relationship. You’re designing Charlie. Let’s talk about where that goes wrong—and how to get it right.

Key Insights

  • Users often do not understand why AI-powered systems request extensive personal data, increasing privacy risks.

  • Trust in AI agents can become excessive, creating new vectors for manipulation by threat actors.

  • Security issues typically occur beneath the surface until alerts disrupt the user experience, often causing frustration.

  • Prompt injection attacks pose a novel threat where malicious inputs manipulate AI agents to access sensitive user data.

  • Multimodal AI interfaces introduce complexity in security decisions, increasing chances for user errors.

  • Secure by default settings reduce burden on users and improve overall protection without requiring user intervention.

  • Cross-disciplinary collaboration between UX, security, product, legal, and compliance teams is crucial for safer AI design.

  • Users need clear, contextual guidance during onboarding to make informed decisions about data sharing and security settings.

  • Transparency about AI limitations and giving users the option to reverse AI actions are essential for building trust.

  • Threat actors are likely to exploit growing AI access to personal data and automate vulnerabilities discovery.

Notable Quotes

"When a product is powered by AI, you're not just designing the features; you are designing an entire relationship."

"Charlie is like the most annoying coworker who constantly surfaces problems but never offers solutions to Alice."

"Threat actors probably know your system better than you do and are looking for any entry points to exploit."

"Alice often perceives Charlie as just another barrage of alerts filled with jargon she doesn't understand."

"Prompt injection attacks can trick AI agents into accessing private data like emails without the user realizing."

"People become incrementally more comfortable giving away data because they see the value AI provides."

"We need secure defaults that protect users out of the box without them having to figure it out."

"Alert fatigue is real; users can't be burdened with constant security decisions or they'll ignore them."

"Giving users the ability to reverse AI-driven actions is critical but currently underexplored."

"If Charlie has been tampered with, Alice needs a clear way to be alerted that she shouldn't trust it."

Ask the Rosenbot
Deanna Zandt
The Unspoken Complexity of “Self-Care” with Deanna Zandt
2022 • Civic Design Community
Steve Portigal
War Stories LIVE! Steve Portigal
2020 • Advancing Research 2020
Gold
Jane Reid
Self-care in User Research
2020 • Advancing Research Community
Bria Alexander
Opening Remarks Day 2
2024 • Advancing Research 2024
Gold
Nicole Aleong
What UX research can learn from other research practices [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Louis Rosenfeld
Coffee with Lou: Should You Write a (UX) Book?
2024 • Rosenfeld Community
Saara Kamppari-Miller
Theme Three Intro
2023 • DesignOps Summit 2023
Gold
Sarah Williams
A Framework for CX Transformation
2021 • Design at Scale 2021
Gold
Aiyana Bodi
Three Key Climate Initiatives and How You Can Help
2024 • Climate UX Interest Group
Adam Cutler
Discussion
2016 • Enterprise UX 2016
Gold
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Civic Design 2022
Gold
Todd Healy
Driving Change with CX Metrics
2023 • Enterprise UX 2023
Gold
Jake Burghardt
Stop wasting research: Unlock more value from research insights
2025 • Rosenfeld Community
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold
Sarah Brooks
Theme 3 Intro
2021 • Civic Design 2021
Gold
Daniela Magaña Flores
Ahead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
2022 • Advancing Research 2022
Gold

More Videos

Bria Alexander

"Our sponsors offer sessions with no sales pitches, just incredibly high-quality content."

Bria Alexander

Opening Remarks

January 8, 2024

Laura Smith

"Planning officers became product owners, gaining confidence in agile and design processes through ongoing involvement."

Laura Smith Tom Gayler

Embedding Service Design and Agile Practice within UK Planning Teams to Create Services that Last

December 3, 2024

Samuel Proulx

"Digital and physical services must be designed as the same thing, not separate products."

Samuel Proulx

Invisible barriers: Why accessible service design can’t be an afterthought

December 3, 2024

Uday Gajendar

"There are certain hushed reverberations ringing through systems thinking."

Uday Gajendar

Theme One Intro

June 6, 2023

Devon Powers

"The future will be struggle; we need to face that forward, not deny it."

Devon Powers

Imagining Better Futures

March 9, 2022

Sam Proulx

"Confidence is crucial — nobody wants to lose money or receive a product they can’t use, so unclear or unlabeled controls cause people with disabilities to give up faster."

Sam Proulx

Online Shopping: Designing an Accessible Experience

November 29, 2023

Joseph Williams

"I was absolutely adamant about not giving work away for free."

Joseph Williams Nepani Birondo Matt Readman Allison Ngo

Unlocking impact and influence through inclusive hiring in research

December 16, 2021

Stephanie Wade

"Moving fast into prototyping helps keep momentum going beyond the fun ideation phase to actual implementation."

Stephanie Wade

Building and Sustaining Design in Government

December 8, 2021

Lona Moore

"Rituals symbolize community identity and shared values, helping scale human-centered habits across boundaries."

Lona Moore

Scaling Design Beyond Designers

June 11, 2021