Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Garbage in, garbage out? Measuring error rates to get ready for AI
Thursday, January 8, 2026 • Rosenfeld Community
Share the love for this talk
Garbage in, garbage out? Measuring error rates to get ready for AI
Speakers: Caroline Jarrett
Link:

Summary

We’re all aware of a big push to implement AI everywhere, including in the services that many of us are working on. It seems only fair to try to give the AI some good quality input in the hope of getting decent output from it. Or, being more pessimistic: we probably expect to get some level of errors from the AI, but what do we know about the error rates in what we’re putting into the AI? In this session, we will compare our ideas on identifying errors and measuring error rates, including thinking about errors in six ways: 1) Problems along the way 2) Wrong result 3) Unnecessary action 4) Delayed-impact problem 5) Non-uptake or over-uptake 6) Technology problem We’ll wrap up with “tips and next steps”: an opportunity to consider what we now need to find out or do differently.

Key Insights

  • Errors in data collection and user input are foundational issues that compromise AI and service outcomes.

  • Users often 'fudge' answers due to ambiguous questions, privacy concerns, or to achieve a desired outcome.

  • Non-uptake, where users abandon a form or process, is a major source of error but is rarely published or measured.

  • Mistakes can be categorized as problems along the way, wrong results, unnecessary actions, and delayed impact issues.

  • Multiple accounts creation often occurs due to users forgetting existing accounts, leading to data duplicates and service inefficiencies.

  • Measuring error rates is complex; different metrics (per person, per attempt, completion vs. start) yield different perspectives.

  • Elections provide a useful model for measuring data quality, using turnout, participation, and eligibility rates.

  • Data quality deteriorates over time due to changes like moving, name changes, loss of documents, or organizational restructuring.

  • AI initiatives can provide a compelling rationale and funding opportunity for improving longstanding data quality problems.

  • Frameworks like the UK Government Data Quality Framework help organizations systematically assess and address data issues.

Notable Quotes

"If we get garbage in, we get garbage out — this is true for AI as much as for surveys or forms."

"People can make all sorts of inventive mistakes on their forms that AI struggles to interpret."

"Sometimes a form forces you into a wrong answer by giving inappropriate options."

"I’ve seen people fudge their date of birth so their child can attend a summer camp they aren’t technically eligible for."

"A major error in many services is users creating multiple accounts because they can’t find or reuse existing ones."

"An error might not be immediate; data can be fine when collected but deteriorate over time and cause problems later."

"Completion rates (conversion rates) and dropout rates are simple metrics but often not tracked or shared."

"Organizations rarely know their error rates, which limits their ability to improve user experience or data accuracy."

"Linking data quality efforts to AI initiatives can help secure attention and budget for necessary improvements."

"Data quality involves accuracy, completeness, uniqueness, timeliness, and representativeness—not just error reduction."

Ask the Rosenbot
Bria Alexander
Opening Remarks
2023 • Advancing Research 2023
Gold
John Cutler
The Alignment Trap
2023 • Design in Product 2023
Gold
Daniel Orbach
Zero to One: Co-Creating Operating Models with your Team
2024 • DesignOps Summit 2024
Gold
Renee Bouwens
Landing Product Impact: Aligning Research as a Foundational Driver for Delivering the World’s Best Products
2023 • QuantQual Interest Group
Bria Alexander
Opening Remarks
2024 • Advancing Research 2021
Gold
Cennydd Bowles
Day 1 Panel
2024 • Designing with AI 2024
Gold
Adrian Howard
Sturgeon’s Biases
2024 • DesignOps Summit 2024
Gold
Aleksandra Korczynska
Survey Tools
2026 • Advancing Research 2026
Conference
Silke Bochat
5 Antifragile Strategies for a DesignOps 2.0
2024 • DesignOps Summit 2024
Gold
Maria Taylor
Knowledge is Power: Managing the Lifeblood of the Design Org
2023 • DesignOps Summit 2023
Gold
Patrizia Bertini
Designing Within the Lines: How the EU AI Act Can Spark Better AI Innovation
2025 • DesignOps Community
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold
Janelle Estes
UX Research Trends
2021 • Advancing Research Community
Sheryl Cababa
Living in the Clouds: Adopting a Systems Thinking Mindset
2023 • Enterprise UX 2023
Gold
Jose Coronado
From Zero to Hero
2022 • DesignOps Summit 2022
Gold

More Videos

Julie Gitlin

"Banks have 144 years of tech baggage making digital-first mindsets hard, and culture change is just as important as technology."

Julie Gitlin Esther Raice

Design as an Agent of Digital Transformation at JPMC

June 9, 2021

John Calhoun

"Hold your destination in mind. Have an intention to succeed. Break up the long tracks and brace for the hard parts."

John Calhoun

Have we Reached Our Peak? Spotting the Next Mountain For DesignOps to Climb

October 1, 2021

Daniel Korczynski

"Human in the loop means constantly interacting with AI, documenting your thoughts and assuring quality."

Daniel Korczynski

Why AI Is Bad at Research (and how to make it actually useful)

March 10, 2026

JJ Kercher

"Now we’ve evolved into customer experience, working horizontally across all silos with service design."

JJ Kercher

A Roadmap for Maturing Design in the Enterprise

June 15, 2018

Emily Lessard

"Long-term contracts and retainer arrangements save time by avoiding constant rebidding."

Emily Lessard

RFPs Without Tears: Writing Inclusive RFPS that Don't Scare Away Talent

December 9, 2021

Kevin Bethune

"Everything is the way it is by design."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Alexandra Schmidt

"Technology moves faster than policymaking — that’s the pacing problem we need to address."

Alexandra Schmidt

Why Ethics Can't Save Tech

November 18, 2022

Dan Hill

"The car is probably one of the most negative technologies we’ve introduced at scale."

Dan Hill

Designing for the infrastructures of everyday life

June 4, 2024

Sam Proulx

"You may need to operate the prototype on behalf of users whose assistive tech can’t click or interact with clickable prototypes."

Sam Proulx

Prototype Reviews, People With Disabilities, and You

December 8, 2021