Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI
Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Rachael Dietkus, LCSW
AI: Passionate defenses and reasoned critique [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Julie Baher
Culture Change—My Journey
2015 • Enterprise UX 2015
Gold
Susan Simon-Daniels
War Stories LIVE! Susan Simon-Daniels
2020 • Advancing Research 2020
Gold
Husani Oakley
Theme Two Intro
2023 • Enterprise UX 2023
Gold
Steve Sanderson
Discussion
2015 • Enterprise UX 2015
Gold
Erin May
Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
2022 • Advancing Research 2022
Gold
Rachael Greene
Building a Design Ops Practice that Really Works (Most of the Time)
2025 • DesignOps Community
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
Elizabeth Sklar
Co-creating research enablement with your tech org: a case study
2026 • Advancing Research 2026
Conference
Joshua Graves
We Need To Talk: Navigating Conversations with Your Boss (Part 1 of 3)
2025 • Rosenfeld Community
Bassel Deeb
Do More With Less: Equip and Lead Design Orgs Through Adversity
2023 • DesignOps Summit 2023
Gold
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Jon Fukuda
Design Planning and Management Support
2023 • DesignOps Summit 2023
Gold
Susan Weinschenk
Evaluating the Maturity of UX in Your Organization
2020 • Enterprise Community
Peter Van Dijck
Designing AI-first products on top of a rapidly evolving technology
2025 • Designing with AI 2025
Gold
Sarah Alvarado
How to make UX research leadership more effective [Advancing Research Community Workshop Series]
2023 • Advancing Research Community

More Videos

Peter Van Dijck

"If a capability exists and is useful, it’s very likely the model will improve significantly on it over time."

Peter Van Dijck

Building impactful AI products for design and product leaders, Part 1: The new product journey

July 16, 2025

JD Buckley

"Ignore the research: it’s ultimately a business decision."

JD Buckley Margot Dear Jim Kalbach Janaki Kumar

COMMUNICATE: Discussion

June 14, 2018

Bria Alexander

"The virtual social event tonight is Cozy Juicy Reel, a fun board game our virtual attendees always love."

Bria Alexander Louis Rosenfeld

Opening Remarks Day 1

March 25, 2024

Joshua Graves

"Our brains are wired for efficiency and use heuristics, so they prefer daydreaming over hard conversations."

Joshua Graves

We Need To Talk: Navigating Conversations with Your Boss (Part 1 of 3)

April 14, 2025

Bria Alexander

"You’re going to get the importance of empathy to the bottom line."

Bria Alexander

Theme Two Intro

September 8, 2022

Joi Freeman

"We lacked the insight to interpret what we were seeing in ways that were culturally relevant in those particular markets."

Joi Freeman

A New Vantage Point: Building a Pipeline for Multifaceted Research(ers)

March 30, 2020

Erin Hauber

"The lawyer got to be the guy who said yes and, not just the guy who said no."

Erin Hauber

Design is Not the Frosting on the Scaled Agile Layer Cake

October 24, 2019

Laura Smith

"Planning officers became product owners, gaining confidence in agile and design processes through ongoing involvement."

Laura Smith Tom Gayler

Embedding Service Design and Agile Practice within UK Planning Teams to Create Services that Last

December 3, 2024

Christian Bason

"We need to innovate how we innovate, expanding cultural and geographic perspectives in design."

Christian Bason

Expand—Rethinking Design for Public Challenges

September 14, 2022