Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Deciding when to automate: Integrating AI in high-stakes systems
This video is featured in the AI Conference playlist.
Summary
In high-stakes primary health systems, designing with AI isn’t just about adding new capabilities. It is about deciding where AI genuinely helps, where it doesn’t, and how to introduce it in contexts where human judgment or public trust are on the line. With many designers raising ethical concerns related to AI, this case study places those risks at the center of a practical design process and offers practical decision tools designers can incorporate into their work.
Key Insights
-
•
Risk in high-stakes AI systems is dynamic and context-dependent, influenced by social trust and local knowledge.
-
•
Introducing AI in complex environments requires a decision gate before ideation to assess feasibility, risks, and appropriateness.
-
•
Designers must define clear boundaries for AI’s role, ensuring human judgment remains central in critical decisions.
-
•
AI fails not because of poor models but due to poor integration into real-world workflows and timing of outputs.
-
•
Collaboration across diverse stakeholders, including frontline workers and health ministries, is vital for responsible AI design.
-
•
Some of the most responsible design decisions involve choosing not to build or deploy certain AI tools.
-
•
Building AI tools on familiar channels like WhatsApp enhances usability and adoption in resource-constrained contexts.
-
•
Validation cycles must assess not only usability but also trust, interpretation, and potential harm of AI outputs.
-
•
AI can effectively support tasks like data synthesis and pattern recognition but should avoid automating final decisions.
-
•
Ethics must serve as a critical filter shaping what AI can and should do within high-stakes systems.
Notable Quotes
"Risk in health and in these communities is not static because it could depend on environmental conditions or social factors."
"We had to stop and ask, should AI be used for this problem space at all?"
"We are no longer just designing usable AI tools. We are also being asked to design key decisions about that AI itself."
"AI doesn’t fail at the model level. It fails at the workflow level."
"Some of the most responsible design decisions are actually about what we do not build."
"It became important that human judgment should remain in control because there were contexts that AI cannot have."
"We built the reporting assistant on WhatsApp because parts of their workflow were already happening informally on such channels."
"Validation is not just about asking whether the tool works. It’s about whether it works safely, realistically, and usefully."
"Design is no longer just about building. It’s also now about setting boundaries."
"The question is not whether AI is useful. It is. The question is what role should AI play."
Or choose a question:
More Videos
"Getting the data out of laptops and onto the walls made the research physical and accessible."
Katy MogalBut Do Your Insights Scale?
March 12, 2021
"Workshops are not just about input, but about diverging and converging to make decisions together."
Shipra KayanMake your research synthesis speedy and more collaborative using a canvas
January 24, 2025
"You have to push against 200,000 years of human evolution to get researchers to create reusable knowledge."
Matt DuignanAtomizing Research: Trend or Trap
March 30, 2020
"When the ground shifts beneath us, we get to decide, do we scramble to regain our footing exactly where we stood before? Or do we take a step in a new direction, explore new terrain, and expand the boundaries of what’s possible?"
Megan BlockerTheme 2 Intro
March 12, 2025
"These AI models are not one hundred percent perfect, so you must stress test the output and watch out for bias."
Kelly DernAI as a Design Partner: How to Get the Most Out of AI Tools to Scale Your Process
October 3, 2023
"Research operations is the people, mechanisms and strategies that set user research in motion."
Fatimah RichmondThe Future of ReOps as a Strategic Function: A Roadmap for Getting There
March 25, 2024
"The system always kicks back—the consequences of our actions unfold over time."
Peter MorvilleThe Architecture of Understanding
May 13, 2015
"Strong product outcomes are about quality of customer understanding and collaboration, not just the artifacts we make."
Jess GrecoCreating a Basis for Change: Scaling Design Maturity
June 8, 2022
"Screen curtain mode isn’t for sighted people simulating blindness; it’s a privacy feature to keep screens safe from prying eyes."
Sam ProulxUnderstanding Screen Readers on Mobile: How And Why to Learn from Native Users
June 6, 2023
