Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

When AI Becomes the User’s Point Person—and Point of Failure

Thursday, August 7, 2025 • Rosenfeld Community
Share the love for this talk
When AI Becomes the User’s Point Person—and Point of Failure
Speakers: Heidi Trost
Link:

Summary

Imagine slipping on a sleek pair of smart glasses. Not only do you look sharp, the glasses capture everything you see, hear, and do. Your AI assistant—built into the glasses and synced to your email, social media accounts, health apps, and finances—manages your life. It’s tasked with paying bills, booking trips, replying to messages, even helping you swipe right. Over time, you find yourself chitchatting with your AI assistant. You call him Charlie. Now imagine you’re a threat actor. That trust between user and AI assistant? It’s your entry point. If your product is powered by AI, you’re not just designing features—you’re designing an entire relationship. You’re designing Charlie. Let’s talk about where that goes wrong—and how to get it right.

Key Insights

  • Users often do not understand why AI-powered systems request extensive personal data, increasing privacy risks.

  • Trust in AI agents can become excessive, creating new vectors for manipulation by threat actors.

  • Security issues typically occur beneath the surface until alerts disrupt the user experience, often causing frustration.

  • Prompt injection attacks pose a novel threat where malicious inputs manipulate AI agents to access sensitive user data.

  • Multimodal AI interfaces introduce complexity in security decisions, increasing chances for user errors.

  • Secure by default settings reduce burden on users and improve overall protection without requiring user intervention.

  • Cross-disciplinary collaboration between UX, security, product, legal, and compliance teams is crucial for safer AI design.

  • Users need clear, contextual guidance during onboarding to make informed decisions about data sharing and security settings.

  • Transparency about AI limitations and giving users the option to reverse AI actions are essential for building trust.

  • Threat actors are likely to exploit growing AI access to personal data and automate vulnerabilities discovery.

Notable Quotes

"When a product is powered by AI, you're not just designing the features; you are designing an entire relationship."

"Charlie is like the most annoying coworker who constantly surfaces problems but never offers solutions to Alice."

"Threat actors probably know your system better than you do and are looking for any entry points to exploit."

"Alice often perceives Charlie as just another barrage of alerts filled with jargon she doesn't understand."

"Prompt injection attacks can trick AI agents into accessing private data like emails without the user realizing."

"People become incrementally more comfortable giving away data because they see the value AI provides."

"We need secure defaults that protect users out of the box without them having to figure it out."

"Alert fatigue is real; users can't be burdened with constant security decisions or they'll ignore them."

"Giving users the ability to reverse AI-driven actions is critical but currently underexplored."

"If Charlie has been tampered with, Alice needs a clear way to be alerted that she shouldn't trust it."

Ask the Rosenbot
B. Pagels-Minor
Breaking the Tension: The Power of Enabling Your Employees to Show Up Authentically
2022 • Design at Scale 2022
Gold
Cornelius Rachieru
Handling Complexity: Framing a Scale of Design
2021 • Design at Scale 2021
Gold
Daniela Magaña Flores
Ahead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
2022 • Advancing Research 2022
Gold
Emily Eagle
Can't Rewind: Radio and Retail
2019 • Enterprise Experience 2019
Gold
Sharon Banh
Reimagining research: What does the field need to grow? [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Prerna Makanawala
Achieving Balanced Design Consistency
2021 • Design at Scale 2021
Gold
Sylvie Abookire
A Civic Designer's Guide to Mindful Conflict Navigation
2022 • Civic Design 2022
Gold
Kelly Goto
Emotion Economy: Ethnography as Corporate Strategy
2015 • Enterprise UX 2015
Gold
Kate Koch
Flex Your Super Powers: When a Design Ops Team Scales to Power CX
2021 • DesignOps Summit 2021
Gold
Amy Brana Stuart
Rest in Peace Fly-in-fly-out Design
2022 • Design at Scale 2022
Gold
Ed Mullen
Designing the Unseen: Enabling Institutions to Build Public Trust
2022 • Civic Design 2022
Gold
Sarah Fathallah
Lessening the Research Burden on Vulnerable Communities
2020 • Advancing Research 2020
Gold
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Husani Oakley
Theme Three Intro
2023 • Enterprise UX 2023
Gold
Marisa Bernstein
It Takes GRIT: Lessons from the Small, but Mighty World of Civic Usability Testing
2021 • Civic Design 2021
Gold
Lena Shenkarenko
Collaborative Wireframing for Creating Team Alignment and Shipping Better Products
2020 • DesignOps Summit 2020
Gold

More Videos

Sam Proulx

"Sometimes accessibility isn’t perfect, but consistency lets me shop on Amazon in my sleep because of muscle memory."

Sam Proulx

Online Shopping: Designing an Accessible Experience

October 3, 2023

Jorge Arango

"Linear notes are fixed hierarchically and chronologically, making it hard to find patterns across ideas."

Jorge Arango

Exploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)

April 19, 2024

Andy Barraclough

"UX research teams spend significant time manually analyzing interviews, limiting capacity to focus on higher-value tasks."

Andy Barraclough Betsy Nelson

From Costly Complexity to Efficient Insights: Why UX Teams Are Switching To Voxpopme

September 23, 2024

John Taschek

"Analysts are paid to call your babies ugly; but many companies hide those reports—I'm pushing all of them up to leadership."

John Taschek

Making People the X-Factor in the Enterprise

June 15, 2018

Jon White

"Deliberate documentation can be the decider between something being useful in the future or not."

Jon White Erin May

Unsticking Research for Better Information Flow

March 11, 2026

Patrick Boehler

"Structure beats free form interaction—systematic query frameworks are essential to avoid pitfalls."

Patrick Boehler

Fishing for Real Needs: Reimagining Journalism Needs with AI

June 10, 2025

Nicole Aleong

"There are over 7 billion futures of right now."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Susan Weinschenk

"You will never get to the point where you don't have to teach UX over and over again because people forget and new people come in."

Susan Weinschenk

Evaluating the Maturity of UX in Your Organization

January 15, 2020

Milan Guenther

"The purpose of a system is what it does—so you can measure an enterprise by its outcomes, not just by its slogans or strategy documents."

Milan Guenther

A Shared Language for Co-Creating Ambitious Endeavours

June 6, 2023