Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Designing inclusively with AI
Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Designing inclusively with AI
Speakers: Kate Kalcevich
Link:

Summary

We know there’s massive opportunity for AI to be used for greater speed, growth, and insights, but how can we avoid the pitfalls? Head of Accessibility Innovation at Fable Kate Kalcevich will share how to best leverage current AI tools in your day-to-day work with inclusion in mind.

Key Insights

  • AI-generated images of people with disabilities often show racial and representation biases, such as all-white subjects and inaccurate assistive device depictions.

  • Disability representation in AI images is frequently inaccurate, e.g., guide dogs shown with leashes instead of harnesses and blindfolds over eyes.

  • AI-generated personas for users with disabilities can contain improbable feature combinations, such as a MacBook user reliant on Windows-only assistive tech like JAWS.

  • Personas created by AI lack the depth and nuance gained from real user research and testing, especially for disability accessibility considerations.

  • AI-generated mockups can have accessibility issues like poor contrast and unlabeled UI elements, requiring human accessibility review.

  • AI tools like Stark Sidekick improve design accessibility through automated issue detection, though they are not yet full replacements for experts.

  • AI-powered task management apps can enhance productivity and accommodation for neurodivergent users by organizing schedules and reducing cognitive load.

  • AI can help identify potential biases in accessibility automation projects, but requires careful prompt crafting and domain expertise.

  • Designing AI-powered interfaces comes with inclusion challenges, as users unfamiliar with prompting may struggle to get effective results.

  • Combining AI output with continuous user testing, especially involving people with disabilities, is essential to ensure accessibility and inclusion.

Notable Quotes

"If you have questions, I will take questions throughout."

"AI-generated images often have no racial diversity and show inaccurate assistive device representation."

"You are either navigating with a white cane or a service dog because you are not gonna want both of your hands full."

"Personas tend to not give you those deep insights that you would get from user research and user testing."

"Automation might not be able to capture the context of an image if you're trying to describe an image."

"AI is actually a pretty good productivity hack as a starting point for accessibility reviews."

"A lot of people will try a prompt and not get what they were hoping to get and maybe miss out on all the benefits of AI."

"For people who identify as autistic, AI tools that review writing tone can be especially helpful."

"If the code doesn’t match the label on screen, voice controls might not recognize the command."

"There always has to be a human option if you’re not getting what you need from the AI."

Ask the Rosenbot
Andy Warr
Under My (Research) Umbrella: The Benefits and Challenges of Building a Unified Insights Function
2024 • Advancing Research 2024
Gold
Heidi Trost
To Protect People, You Have to Protect Information: A Human-Centered Design Approach to Cybersecurity
2025 • Rosenfeld Community
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Design in Product 2023
Gold
Chris Chapo
Data Science and Design: A Tale of Two Tribes
2015 • Enterprise UX 2015
Gold
Peter Morville
The Architecture of Understanding
2015 • Enterprise UX 2015
Gold
Anne Mamaghani
How Your Organization's Generative Workshops Are Probably Going Wrong and How to Get Them Right
2023 • Advancing Research 2023
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • DesignOps Summit 2025
Gold
Luca Rager
Empowering Gaming at Scale: How Xbox Builds Powerful, Automated, and Distributed Design Systems with Sketch
2021 • DesignOps Summit 2021
Gold
Tristin Oldani
Turning awareness into action with Climate UX
2025 • Climate UX Interest Group
Maria Giudice
Remaking the Making Company: Moving from Product to Experience
2016 • Enterprise UX 2016
Gold
Dan Hill
Strategic design, slowdown, and the infrastructures of everyday life
2022 • Enterprise Community
Steve Sanderson
Discussion
2015 • Enterprise UX 2015
Gold
Marieke McCloskey
User Science: Product Analytics & User Research
2021 • Advancing Research 2021
Gold
Dantley Davis
Leadership & Diversity—A Fireside Chat with Dantley Davis
2020 • Enterprise Community
Abbey Smalley
Scaling UX Past the Size of Your Team
2024 • Enterprise Experience 2020
Gold
Lisa Gironda
Opener: Chief of Staff–An unexpected journey
2024 • DesignOps Summit 2020
Gold

More Videos

Joerg Beringer

"The system is live now analyzing audience-submitted tasks and producing results in minutes."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

September 10, 2025

Jennifer Kong

"Documentation is important for liability and useful for patient handoffs, so notes must be concise yet detailed."

Jennifer Kong

Journeying toward AI-assisted documentation in healthcare

June 5, 2024

Dr. Jamika D. Burge

"There’s this inflection point about how research work is done and what the system wants from knowledge workers."

Dr. Jamika D. Burge Steve Portigal Alba Villamil Sam Ladner

The Future of Research: Bridging the Gaps

July 29, 2021

Steve Krug

"It’s not self-driving cars, it’s power steering."

Steve Krug Louis Rosenfeld

Don’t Make Me Think 3.0: What Endures and What Evolves in UX

March 11, 2026

Zen Ren

"Using cognitive dimensions to analyze findings helps clarify which recommendations to make and keeps us in the real problem context."

Zen Ren

Taking Inspiration from Instructional Design for Research

March 10, 2022

Ariel Kennan

"Many governments need to buy design services to augment their internal capacities."

Ariel Kennan

Theme 2 Intro

December 9, 2021

Cennydd Bowles

"People remember how you made them feel, and that’s something AI can fake but not truly replicate."

Cennydd Bowles Dan Rosenberg Lisa Welchman

Day 1 Panel

June 4, 2024

Alana Washington

"I knew that probably 80% of the problem was about teamwork and maybe 20% was about stuff we needed to build."

Alana Washington

(Remote) Service Design: A Transformation Case Study

June 8, 2022

Allison Sanders

"If your purpose is just getting through 2020, that’s okay — survival can be the goal right now."

Allison Sanders

Operating with Purpose

January 8, 2024