Deciding when to automate: Integrating AI in high-stakes systems
This video is featured in the AI Conference playlist.
Summary
Key Insights
-
•
Risk in high-stakes AI systems is dynamic and context-dependent, influenced by social trust and local knowledge.
-
•
Introducing AI in complex environments requires a decision gate before ideation to assess feasibility, risks, and appropriateness.
-
•
Designers must define clear boundaries for AI’s role, ensuring human judgment remains central in critical decisions.
-
•
AI fails not because of poor models but due to poor integration into real-world workflows and timing of outputs.
-
•
Collaboration across diverse stakeholders, including frontline workers and health ministries, is vital for responsible AI design.
-
•
Some of the most responsible design decisions involve choosing not to build or deploy certain AI tools.
-
•
Building AI tools on familiar channels like WhatsApp enhances usability and adoption in resource-constrained contexts.
-
•
Validation cycles must assess not only usability but also trust, interpretation, and potential harm of AI outputs.
-
•
AI can effectively support tasks like data synthesis and pattern recognition but should avoid automating final decisions.
-
•
Ethics must serve as a critical filter shaping what AI can and should do within high-stakes systems.
Notable Quotes
"Risk in health and in these communities is not static because it could depend on environmental conditions or social factors."
"We had to stop and ask, should AI be used for this problem space at all?"
"We are no longer just designing usable AI tools. We are also being asked to design key decisions about that AI itself."
"AI doesn’t fail at the model level. It fails at the workflow level."
"Some of the most responsible design decisions are actually about what we do not build."
"It became important that human judgment should remain in control because there were contexts that AI cannot have."
"We built the reporting assistant on WhatsApp because parts of their workflow were already happening informally on such channels."
"Validation is not just about asking whether the tool works. It’s about whether it works safely, realistically, and usefully."
"Design is no longer just about building. It’s also now about setting boundaries."
"The question is not whether AI is useful. It is. The question is what role should AI play."
Or choose a question:
More Videos
"An elite athlete doesn’t turn up at the Olympics and do their thing without coaches, physios, nutritionists."
Dr. Karl JeffriesThe Science of Creativity for DesignOps
January 8, 2024
"Investment is attention: If the business funds you, it means they believe in what you do."
Amy MarquezINVEST: Discussion
June 15, 2018
"Both tribes really believe in doing awesome for customers, that’s our common ground."
Steve Portigal Chris Chapo Kelly Goto Christian RohrerDiscussion
May 13, 2015
"Constant communication is necessary; agreement emails can’t be one-and-done."
Alicia MootyDesign Staffing Models
September 30, 2021
"UX should be a critical dependency, not an afterthought or checkbox on the to-do list."
Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt LaurenceThe Authentic UX Talent Show
January 8, 2024
"Junior team members don’t have to talk directly to executives; working effectively with peers can create grassroots buy-in."
Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne RogondinoTheme 1: Discussion
January 8, 2024
"We made it a priority from day one to support data exports so customers can leave without wasting weeks migrating data."
Sofia QuinteroThe Product Philosophy Behind EnjoyHQ
March 10, 2021
"You are not the first person to try something like this, and they’re typically not looking forward to the next transformation."
Laura KleinUnique challenges of innovation in enterprises
April 23, 2020
"You’ve got to take control of the backlog and position your team as value-driven, not just servicing individual teams."
Andrew MichaelBuilding a Product Insights Team
March 10, 2022
