Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Deciding when to automate: Integrating AI in high-stakes systems
Summary
In high-stakes primary health systems, designing with AI isn’t just about adding new capabilities. It is about deciding where AI genuinely helps, where it doesn’t, and how to introduce it in contexts where human judgment or public trust are on the line. With many designers raising ethical concerns related to AI, this case study places those risks at the center of a practical design process and offers practical decision tools designers can incorporate into their work.
Key Insights
-
•
Risk in high-stakes AI systems is dynamic and context-dependent, influenced by social trust and local knowledge.
-
•
Introducing AI in complex environments requires a decision gate before ideation to assess feasibility, risks, and appropriateness.
-
•
Designers must define clear boundaries for AI’s role, ensuring human judgment remains central in critical decisions.
-
•
AI fails not because of poor models but due to poor integration into real-world workflows and timing of outputs.
-
•
Collaboration across diverse stakeholders, including frontline workers and health ministries, is vital for responsible AI design.
-
•
Some of the most responsible design decisions involve choosing not to build or deploy certain AI tools.
-
•
Building AI tools on familiar channels like WhatsApp enhances usability and adoption in resource-constrained contexts.
-
•
Validation cycles must assess not only usability but also trust, interpretation, and potential harm of AI outputs.
-
•
AI can effectively support tasks like data synthesis and pattern recognition but should avoid automating final decisions.
-
•
Ethics must serve as a critical filter shaping what AI can and should do within high-stakes systems.
Notable Quotes
"Risk in health and in these communities is not static because it could depend on environmental conditions or social factors."
"We had to stop and ask, should AI be used for this problem space at all?"
"We are no longer just designing usable AI tools. We are also being asked to design key decisions about that AI itself."
"AI doesn’t fail at the model level. It fails at the workflow level."
"Some of the most responsible design decisions are actually about what we do not build."
"It became important that human judgment should remain in control because there were contexts that AI cannot have."
"We built the reporting assistant on WhatsApp because parts of their workflow were already happening informally on such channels."
"Validation is not just about asking whether the tool works. It’s about whether it works safely, realistically, and usefully."
"Design is no longer just about building. It’s also now about setting boundaries."
"The question is not whether AI is useful. It is. The question is what role should AI play."
Or choose a question:
More Videos
"Scaffolding is the temporary structure that provides support, safety, and access while the real work happens underneath."
Bianca JeffersonFrom Sprints to Systems: Operationalizing Continuous Discovery Through DesignOps
September 10, 2025
"Lifting your game above the day to day is impossible without ops."
Brigette MetzlerScaling ResearchOps: Helping Researchers do Their Best Work
March 30, 2020
"AI moderation, in short, is a large language model that runs a moderated interview session without a human researcher present."
Tara TresselInvestigating qualitative depth of AI-moderated interviews
March 10, 2026
"The biggest myth of all is you can't teach creativity."
Dr. Karl JeffriesThe Science of Creativity for DesignOps
January 8, 2024
"Whenever you make a process model or service blueprint, make a wrong model first, then let the experts correct it to build shared understanding."
Milan GuentherA Shared Language for Co-Creating Ambitious Endeavours
June 6, 2023
"True sustainability work requires cross-functional collaboration among product, design, engineering, and sustainability offices."
Chris HammondEmbedding sustainability into enterprise design and development: A journey towards "sustainability consciousness"
April 2, 2025
"You cannot brute force your way into engagement; it just doesn’t happen."
Sofia QuinteroBeyond Tools: The Messy Business of Implementing Research Repositories
March 10, 2022
"You can’t heal your way out of death or oppression by reforming oppressive systems; you must dismantle them."
Uday Gajendar Rachael Dietkus, LCSW Dr. Dawn Emerick Dawn E. Shedrick, LCSWLeading through the long tail of trauma
July 7, 2022
"Dimensionality reduction algorithms take data points in high dimensional space and put them close together in 2D if they're similar."
Ian JohnsonLatent Scope: Finding structure in unstructured data
June 11, 2025
