Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.
Key Insights
-
•
Errors in data collection and user input are foundational issues that compromise AI and service outcomes.
-
•
Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.
-
•
Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.
-
•
Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.
-
•
Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.
-
•
Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.
-
•
Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.
-
•
Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.
-
•
AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.
-
•
Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.
Notable Quotes
"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."
"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."
"Sometimes a form forces you into a wrong answer by giving inappropriate options."
"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."
"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."
"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."
"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."
"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."
"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."
"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."
Or choose a question:
More Videos
"You can’t lead without being a lifelong learner because when your company stops learning, it dies."
Louis RosenfeldDiscussion: What Operations can teach DesignOps
November 6, 2017
"There is still so much work to be done: from dealing with climate change, protecting wildlife, rebuilding infrastructure, feeding the world, ending disease."
Janaki KumarInnovate with Purpose
June 14, 2018
"Designers can be first responders, not just in digital but in real disaster situations."
Lada GorlenkoTheme 2 Intro
June 9, 2022
"Embedding community members to lead design sessions helps shift power dynamics and surfaces authentic voices."
Justin Entzminger Terrance Smith Tracy M. Colunga Mai-Ling GarciaRisk and Reward: How to Diversify the Field of Civic Innovators and Designers
November 17, 2022
"Flexibility is the network's ability to reconfigure itself and yet retain its goals; scalability means expanding or shrinking size with little disruption; survivability means withstanding attacks to nodes or codes."
Gordon Ross12 Months of COVID-19 Design and Digital Response with the British Columbia Government
December 8, 2021
"If the folks who design LLMs don’t know exactly how they work, it’s okay if you don’t either."
Jonathan Fairman Kevin JohnsonIntegrating generative AI into enterprise products: A case study from dscout
June 5, 2024
"Power dynamics exist in every session. People don’t want to be embarrassed or feel put on the spot."
Mila Kuznetsova Lucy DentonHow Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices
March 9, 2022
"When I talk to stakeholders, I try to understand how they are measured and what success means for their role in the company."
Toby HaugDiscussion
June 9, 2017
"Even if there is an element of enrichment in research participation, all processes have some extraction and must be minimized."
Rachael Dietkus, LCSW Uday Gajendar Dr. Dawn Emerick Dawn E. Shedrick, LCSWLeading through the long tail of trauma
July 7, 2022
