Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Garbage in, garbage out? Measuring error rates to get ready for AI
Summary
Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI?
In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways:
1) Problems along the way
2) Wrong result
3) Unnecessary action
4) Delayed-impact problem
5) Non-uptake or over-uptake
6) Technology problem
We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.
Key Insights
-
•
Errors in data collection and user input are foundational issues that compromise AI and service outcomes.
-
•
Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.
-
•
Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.
-
•
Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.
-
•
Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.
-
•
Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.
-
•
Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.
-
•
Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.
-
•
AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.
-
•
Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.
Notable Quotes
"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."
"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."
"Sometimes a form forces you into a wrong answer by giving inappropriate options."
"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."
"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."
"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."
"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."
"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."
"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."
"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."
Or choose a question:
More Videos
"Snow clearing policies choose to clear roads before pavements, deeply ignoring responsibilities that are likely to fall on women."
Mansi GuptaDrawing from Feminist Practice to Make Inclusive Design Operational
September 9, 2022
"Purple gradients are the equivalent of AI slop from 2022 — not today's AI slop."
Paul BakausKilling the handoff: Iterating design live in the browser on real production code
June 10, 2026
"We break down overarching messages into pillar messages and key proof points to create digestible, empathetic communication."
Megan Nipe Lyndsay BoothHuman-Centered Design for Engagement: Maturing from Newsletterville to Personalized, One-to-One Messaging
December 8, 2021
"I started spending more time understanding agent processes at a deeper level beyond just answering tactical questions."
Kayla Farrell Chelsey Glasson Sean Fitzell Jared LeClercWhat It's Like To Be a User Researcher at Compass
March 12, 2021
"Show don’t tell is really critical for getting stakeholder buy-in on a new tool like this."
Michelle Bejian Lotia Anne-Marie MorellRolling Out a Repository: How Zapier Centralizes Insights from Across their Organization
March 28, 2023
"We can all become experts at stuff if we work hard enough – sometimes you have to fake it till you make it."
Megan BlockerGetting to the “So What?”: How Management Consulting Practices Can Transform Your Approach to Research
March 26, 2024
"The metaverse is the successor to the mobile internet, and we’re only at the dial-up stage of its development."
Kate SternScaling Learning for the Future
September 9, 2022
"Feedback is a lot like holding up a mirror. You're reflecting what's already working and like using a flashlight to hide in the corners."
Vanessa VarinFeedback: The Other F-Word
September 10, 2025
"Sponsor sessions are so great that people want recordings. They’re not sales pitches."
Uday Gajendar Louis RosenfeldDay 2 Welcome
June 5, 2024
