Rosenverse
Managing the Human Engagement Risks of AI
Gold
Tuesday, June 10, 2025 • Designing with AI 2025

This video is featured in the AI and UX playlist and 1 more.

Share the love for this talk
Managing the Human Engagement Risks of AI
Speakers: Ovetta Sampson
Link:

Summary

AI is fast transforming the way companies operate and design products and services. New design practices like “vibe coding,” or “prompt engineering,” are accelerating product launches at speed and scale. But with the rush to market; how can teams maintain their commitment to human-centered design, minimize the risks to consumers and keep up with the increase in developer velocity? We showcase various tactics companies are using to manage the risk of implementing AI into their products while keeping their ever tightening GTM timelines.

Key Insights

  • Generative AI does not think or understand; it excels at pattern recognition and filling in blanks like a souped-up mad lib.

  • AI design requires a paradigm shift from building products to designing dynamic behaviors within multi-agent ecosystems.

  • UX designers must move beyond traditional user-device interaction design to address machine learning's stochastic and contextual outputs.

  • Generative AI models have no moral code and can generate harmful outputs without human-guided guardrails and feedback loops.

  • Red teaming, adversarial testing, and safety classifiers are essential tools to proactively identify and mitigate AI risks.

  • Data and model cards documenting dataset provenance and model training are crucial for auditing bias and risk in AI systems.

  • Automation bias can cause humans to overtrust AI outputs, requiring design strategies that clearly signal uncertainty and keep users in control.

  • Machine learning models can unlearn harmful behaviors given correct feedback, but timely monitoring is critical before embedded biases solidify.

  • Mindful AI design entails co-design, simulations of multiple futures, and iterative testing with diverse users to understand AI behavior in context.

  • UX professionals hold unique responsibility and agency to embed ethical, equitable practices in AI design and deployment.

Notable Quotes

"Generative AI is a souped up mad lib, its job is to fill in the blank."

"Good UX and good design need not only to make a comeback but lead the way in this era of AI."

"In the post software era, we have to think about how products will behave, not just how they are built."

"The user and machine can both be in control; technology affordance changes over time and outcomes are dynamic."

"Generative AI has no moral code; you have to discipline it so it can learn right from wrong."

"There is no unbiased creation of machine learning or artificial intelligence; the idea of agnostic algorithms is a myth."

"You have to build guardrails and test models with red teaming to ensure they behave ethically and safely."

"Minimize anthropomorphism in AI design so users know they are engaging with machines, not humans."

"Machine learning models can unlearn behaviors, but we need robust feedback loops to detect when they go awry."

"The more that you engage with AI technology, the better confidence you will have in designing with it."

Ask the Rosenbot
Brendan Jarvis
Framing Tomorrow by Questioning Today
2022 • Design at Scale 2022
Gold
Xenia Adjoubei
Empowering Communities Through the Researcher in Residence Program
2023 • Advancing Research 2023
Gold
Tutti Taygerly
Videconference: How to Work with Difficult People with Tutti Taygerly
2020 • Enterprise Community
Christian Crumlish
The Pygmalion Effect: In Which a Vibe Coding Experiment Becomes a Million Lines…
2025 • Rosenfeld Community
Heidi Trost
When AI Becomes the User’s Point Person—and Point of Failure
2025 • Rosenfeld Community
Ted Booth
Discussion
2016 • Enterprise UX 2016
Gold
Sheri Byrne-Haber
Accessibility at Scale
2021 • Design at Scale 2021
Gold
Liam Thurston
Why Your Design Team Is Quitting, And How To Fix It
2022 • Design at Scale 2022
Gold
Aaron Stienstra
Leveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
2021 • Civic Design 2021
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold
Julie Norvaisas
Back to basics, or start from scratch?
2025 • Advancing Research 2025
Gold
Jane Davis
Strategic Shifts and Innovations in User Research: Navigating Challenges and Opportunities
2025 • Advancing Research 2025
Gold
Helen Armstrong
Augment the Human. Interrogate the System.
2023 • Enterprise UX 2023
Gold
PJ Buddhari
Meet Spectrum, Adobe’s Design System
2021 • Design at Scale 2021
Gold
Theresa Marwah
How Atlassian is Operationalizing Respect in Research
2020 • Advancing Research Community
Mariah Hay
BUILD: Discussion
2018 • Enterprise Experience 2018
Gold

More Videos

Gabrielle Verderber

"Avoid walls of text and create visual hierarchy to ensure scanability by using headers, sections, lists, and images."

Gabrielle Verderber

Documentation Your Team Will Actually Use

October 3, 2023

Jennifer Kanyamibwa

"If you really want to build culture and see how people cross collaborate, try to make lunch with your entire team every single week."

Jennifer Kanyamibwa

Creating the Blueprint: Growing and Building Design Teams

November 8, 2018

Saara Kamppari-Miller

"Our KSPs do not change year over year because changing them frequently is disruptive for the business direction."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Samuel Proulx

"Testing with a screen reader is fine for QA, but user research must be with actual screen reader users."

Samuel Proulx

Designing beyond caricatures: Embracing real, diverse user needs

December 4, 2024

Catherine Dubut

"The store mobile device is the indispensable silent partner enabling store employees to balance task productivity and customer engagement."

Catherine Dubut

Bridging Physical and Digital Spaces: Approaches to Retail Service Design

March 18, 2021

Sam Ladner

"Enterprise user experiences are often customer experiences, not user experiences."

Sam Ladner

Data Exhaust and Personal Data: Learning from Consumer Products to Enhance Enterprise UX

June 8, 2016

Sam Proulx

"When you magnify prototypes, they may look blurry, but that’s something to fix in the final website design."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021

Jemma Ahmed

"Participants want to take ownership of their stories and demand participatory, non-extractive, trauma-informed research methods."

Jemma Ahmed Megan Blocker Eduardo Ortiz

Redefining the research toolkit: Expanding methodologies for a changing world

March 12, 2025

Kevin Bethune

"I can’t tell you how many times recruiters ask, can I have a beer with this person, instead of focusing on true inclusion."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022