Accessible only to conference ticket holders.
Log in Create account
For 90 days after a conference, only paid ticket holders can watch conference videos. After that, all Gold members have access.
If you have purchased recording access and cannot see the video, please contact support.
Deciding when to automate: Integrating AI in high-stakes systems
Summary
In high-stakes primary health systems, designing with AI isn’t just about adding new capabilities. It is about deciding where AI genuinely helps, where it doesn’t, and how to introduce it in contexts where human judgment or public trust are on the line. With many designers raising ethical concerns related to AI, this case study places those risks at the center of a practical design process and offers practical decision tools designers can incorporate into their work.
Key Insights
-
•
Risk in high-stakes AI systems is dynamic and context-dependent, influenced by social trust and local knowledge.
-
•
Introducing AI in complex environments requires a decision gate before ideation to assess feasibility, risks, and appropriateness.
-
•
Designers must define clear boundaries for AI’s role, ensuring human judgment remains central in critical decisions.
-
•
AI fails not because of poor models but due to poor integration into real-world workflows and timing of outputs.
-
•
Collaboration across diverse stakeholders, including frontline workers and health ministries, is vital for responsible AI design.
-
•
Some of the most responsible design decisions involve choosing not to build or deploy certain AI tools.
-
•
Building AI tools on familiar channels like WhatsApp enhances usability and adoption in resource-constrained contexts.
-
•
Validation cycles must assess not only usability but also trust, interpretation, and potential harm of AI outputs.
-
•
AI can effectively support tasks like data synthesis and pattern recognition but should avoid automating final decisions.
-
•
Ethics must serve as a critical filter shaping what AI can and should do within high-stakes systems.
Notable Quotes
"Risk in health and in these communities is not static because it could depend on environmental conditions or social factors."
"We had to stop and ask, should AI be used for this problem space at all?"
"We are no longer just designing usable AI tools. We are also being asked to design key decisions about that AI itself."
"AI doesn’t fail at the model level. It fails at the workflow level."
"Some of the most responsible design decisions are actually about what we do not build."
"It became important that human judgment should remain in control because there were contexts that AI cannot have."
"We built the reporting assistant on WhatsApp because parts of their workflow were already happening informally on such channels."
"Validation is not just about asking whether the tool works. It’s about whether it works safely, realistically, and usefully."
"Design is no longer just about building. It’s also now about setting boundaries."
"The question is not whether AI is useful. It is. The question is what role should AI play."
Or choose a question:
More Videos
"Research was completely evaluative and tactical, positioned as serving design by validating wireframes and high fidelity prototypes."
Johanna KollmannInsights-Driven Product Strategy: Get your Research to Count
December 6, 2022
"Having agents as part of the team makes customer involvement much quicker and easier."
Scher Foord Corey Greenltch Sarah RoweTurn the Ship Around: How to Apply Design Thinking Across Your Organization
June 10, 2021
"It’s all too easy to navigate to a mode of protection and become the gatekeeper controlling what is let in and out of the team."
Kevin BethuneGatekeepers and Servant Leadership
January 30, 2020
"You cannot provide end-to-end service for every researcher, but you can deliver highly supported self-service — Kate Towsey."
Kate TowseyThe State of ResearchOps: More Than Just Theory
June 20, 2019
"How do they grow and organize their teams to keep up? How do they scale their processes? How do they maximize efficiency and quality?"
Jemma AhmedTheme 2 Intro
January 8, 2024
"From novice to intermediate design fluency offers enormous returns, but going from intermediate to expert is a massive investment with negligible return."
Andrew WebsterScaling Design Capability: How Involved Should You Be?
September 30, 2021
"The conference is for mature, experienced designers and managers who connect the dots across complexity and scale."
Wendy Johansson Surya VankaDesign at Scale: Behind the Scenes
April 29, 2021
"I never want to walk into a review with my stakeholders at the SVP level and see something for the first time."
Amy MarquezINVEST: Discussion
June 15, 2018
"If you are not speaking to the CEO or C-executives about compliance and transformation, someone else will."
Patrizia BertiniThe (r)evolution of designOps: It’s Time to Think (really) BIG
September 11, 2025
