Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Deciding when to automate: Integrating AI in high-stakes systems
Summary
In high-stakes primary health systems, designing with AI isn’t just about adding new capabilities. It is about deciding where AI genuinely helps, where it doesn’t, and how to introduce it in contexts where human judgment or public trust are on the line. With many designers raising ethical concerns related to AI, this case study places those risks at the center of a practical design process and offers practical decision tools designers can incorporate into their work.
Key Insights
-
•
Risk in high-stakes AI systems is dynamic and context-dependent, influenced by social trust and local knowledge.
-
•
Introducing AI in complex environments requires a decision gate before ideation to assess feasibility, risks, and appropriateness.
-
•
Designers must define clear boundaries for AI’s role, ensuring human judgment remains central in critical decisions.
-
•
AI fails not because of poor models but due to poor integration into real-world workflows and timing of outputs.
-
•
Collaboration across diverse stakeholders, including frontline workers and health ministries, is vital for responsible AI design.
-
•
Some of the most responsible design decisions involve choosing not to build or deploy certain AI tools.
-
•
Building AI tools on familiar channels like WhatsApp enhances usability and adoption in resource-constrained contexts.
-
•
Validation cycles must assess not only usability but also trust, interpretation, and potential harm of AI outputs.
-
•
AI can effectively support tasks like data synthesis and pattern recognition but should avoid automating final decisions.
-
•
Ethics must serve as a critical filter shaping what AI can and should do within high-stakes systems.
Notable Quotes
"Risk in health and in these communities is not static because it could depend on environmental conditions or social factors."
"We had to stop and ask, should AI be used for this problem space at all?"
"We are no longer just designing usable AI tools. We are also being asked to design key decisions about that AI itself."
"AI doesn’t fail at the model level. It fails at the workflow level."
"Some of the most responsible design decisions are actually about what we do not build."
"It became important that human judgment should remain in control because there were contexts that AI cannot have."
"We built the reporting assistant on WhatsApp because parts of their workflow were already happening informally on such channels."
"Validation is not just about asking whether the tool works. It’s about whether it works safely, realistically, and usefully."
"Design is no longer just about building. It’s also now about setting boundaries."
"The question is not whether AI is useful. It is. The question is what role should AI play."
Or choose a question:
More Videos
"A lot of people think 95% confidence is the only way, but you can adjust confidence levels based on your situation and needs."
Caroline Jarrett Erin WeigelHave fun with statistics?
December 12, 2024
"Mural is not just a digital whiteboard — it is the digital office space where collaboration happens across time and space."
Erika FlowersIntroduction to MURAL for UX
June 11, 2021
"Designing for the average means designing for nobody."
Samuel ProulxFrom Standards to Innovation: Why Inclusive Design Wins
November 19, 2025
"If the code doesn’t match the label on screen, voice controls might not recognize the command."
Kate KalcevichDesigning inclusively with AI
June 5, 2024
"Being present in meetings not led by design and listening can be more powerful than leading every conversation."
Carol MassaDesigning Health: Integrating Service Design, Technology, and Strategy to Transform Patient and Clinician Experiences
December 3, 2024
"Team leadership means managing down to empower your team, managing across to build stakeholder relationships, and managing up with executives."
Peter MerholzCustomer-Centered Design Organizations
June 8, 2017
"We want to know agents better than they know themselves to drive tailored, contextually relevant solutions."
Greg PetroffThe Compass Mission
March 10, 2021
"Working from home is fun only when you can actually leave your home; during a pandemic, it’s incredibly hard."
Tricia WangSpatial Collapse: Designing for Emergent Culture
January 8, 2024
"Design school’s mission is to connect needs of the design organization with learning experiences that impact product quality and careers."
Kate SternScaling Learning for the Future
September 9, 2022
