Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
Friday, February 18, 2022 • QuantQual Interest Group
Share the love for this talk
Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
Speakers: Brian T. O’Neill , Maria Cipollone , Luis Colin , Manuel Dahm and Mike Oren
Link:

Summary

Join us for a different type of Quant vs. Qual discussion: instead of discussing how data science and quantitative research methods can power UX research and design, we’re going to talk about designing enterprise data products and tools that put ML and analytics into the hands of users. Does this call for new, different, or modified approaches to UX research and design? Or do these technologies have nothing to do with how we approach design for data products? The session’s host will be Brian T. O’Neill who is also the host of the Experiencing Data podcast and founder of Designing for Analytics, an independent consultancy that helps data products leaders use design-driven innovation to deliver better ML and analytics user experiences. In this session, we’ll be sharing some rapid [slides-optional] anecdotes and stories from the attendees and then open up the conversation to everyone. We hope to get perspectives from both enterprise data teams doing “internal” data analytics or ML/AI solutions development as well as software/tech companies as well who may offer data-related platform tools, intelligence/SAAS products, BI/decision support solutions, etc. Slots are open to both experienced UX practitioners as well as data science / analytics / technical participants who may have participated in design or UX work with colleagues. Please share! If folks are too quiet in the session, you may be subject to a drum or tambourine solo from Brian. Nobody has all of this “figured out yet” and experiments and trials are welcome.

Key Insights

  • UX researchers in ML-heavy environments often face steep domain knowledge gaps requiring strong interview facilitation to bridge communication.

  • Designing for AI data products demands a shift from generic user interfaces to tools tailored for expert users who need explainability over polish.

  • Cross-functional collaboration thrives by creating shared spaces that respect differing expertise while focusing on a unifying product goal.

  • Decision culture focusing on how decisions are made is more useful than a vague data culture mindset when designing AI/ML products.

  • Besides customers and stakeholders, data labelers who curate training data form a critical, often overlooked human part of the ML ecosystem.

  • Model interpretability can be more important than raw accuracy because users must trust a system to adopt it and generate value.

  • Prototyping data products requires believable data and can be facilitated by testing edge cases like false positives and trust thresholds early.

  • Incremental learning loops involving user feedback can improve models over time and should be designed into AI products.

  • Contextualizing data model outputs with business rules and user environment (e.g., seasonal factors) increases relevance and trust.

  • No-code data science tools and scenario building help teams quickly experiment with data products despite the usual slow model training cycles.

Notable Quotes

"Smoke comes out of my ears when I'm talking to some of the more advanced applications in machine learning."

"Design is about creating that shared space where we look at problems through different lenses."

"Decision culture is a better lens than data culture for thinking about what decisions we're trying to facilitate with AI."

"Not everyone using machine learning is your user; think also about the people labeling the data feeding these models."

"If nobody uses this because they don’t trust it, it doesn’t matter how accurate the model is."

"You need to prototype with real or believable data so the data doesn’t change the context of use."

"Some things that look like blockers, like domain expertise gaps, actually force you to become a better interviewer."

"We’re not always trying to use machine learning everywhere—sometimes the answer is to ignore it."

"Users want to understand why this happened, what will happen, and how can we make it happen—explainability is critical."

"The presenting problem might be a dashboard with a score, but the real need is deciding what to do with that information."

Ask the Rosenbot
Crystal Philcox
The Many Faces of Operations
2017 • DesignOps Summit 2017
Gold
Chris Moses
Stretching the Definition of DesignOps with Product Development
2018 • DesignOps Summit 2018
Gold
Fatimah Richmond
The Future of ReOps as a Strategic Function: A Roadmap for Getting There
2024 • Advancing Research 2024
Gold
Jon Fukuda
Theme One Intro
2023 • DesignOps Summit 2023
Gold
Laura Schaefer
DesignOps: A Conduit for Inclusion
2022 • DesignOps Summit 2022
Gold
Andrew Webster
Scaling Design Capability: How Involved Should You Be?
2021 • DesignOps Summit 2021
Gold
Kevin Bethune
Reimagining Design: Unlocking Strategic Innovation
2022 • Design at Scale 2022
Gold
Jorge Arango
[Demo] How to re-categorize content at scale using LLMs
2024 • Designing with AI 2024
Gold
Louis Rosenfeld
GenAI for UXers: A Rosenbot Demo and Discussion
2025 • Rosenfeld Community
Cennydd Bowles
Responsible Design in Reality
2021 • Design at Scale 2021
Gold
Cheryl Platz
Merging Improv with Design
2019 • Enterprise Community
Dave Malouf
The Past, Present, and Future of DesignOps: a 2-part DesignOps Community Call (Part 2)
2022 • DesignOps Community
Dan Willis
Filling the Void
2018 • DesignOps Summit 2018
Gold
Prerna Makanawala
Achieving Balanced Design Consistency
2021 • Design at Scale 2021
Gold
Llewyn Paine
[Demo] Deploying AI doppelgangers to de-identify user research recordings
2024 • Designing with AI 2024
Gold
Chris Chapo
Data Science and Design: A Tale of Two Tribes
2015 • Enterprise UX 2015
Gold

More Videos

Sheryl Cababa

"Systems thinking can help us design more explicitly for resilience, as seen in distributed manufacturing of PPE during the pandemic."

Sheryl Cababa

Expanding Your Design Lens with Systems Thinking

February 23, 2023

Caitlyn Hampton

"The agents are brutally honest with their feedback, which helps us prioritize features that actually matter to them."

Caitlyn Hampton Monica Lee Jina Yoon

Compass 101: Growing Your Career In A Startup World

June 11, 2021

Silke Bochat

"Please don’t delegate hiring to HR—hiring managers must set clear criteria and be personally involved to avoid bias."

Silke Bochat

5 Antifragile Strategies for a DesignOps 2.0

September 23, 2024

Christian Rohrer

"Security and great user experience is almost a null set — we need to enlarge that circle."

Christian Rohrer

Insight Types That Influence Enterprise Decision Makers

May 13, 2015

Jaime Creixems

"Leave some space for creativity in your design system so designers can innovate without breaking consistency."

Jaime Creixems

Best Practices when Creating and Maintaining a Design System

June 7, 2023

Jeff Gothelf

"Disruptive innovation is about changing customer behavior, not shipping features."

Jeff Gothelf

Innovation Studios: the Engines of Enterprise Experimentation

May 14, 2015

Josh Clark

"Version vaults save multiple versions of work, letting users review, compare, and revert, which encourages exploration."

Josh Clark Veronika Kindred

Sentient Design: New Design Patterns for New Experiences (3rd of 3 seminars)

February 12, 2025

Sarah Rink

"Don’t do remote contextual inquiry just because you can; you lose so much context."

Sarah Rink

Remote User Research: Dos and Don'ts from the Virtual Field

June 11, 2020

Alexandra Schmidt

"Facilitation is the grease between the wheels of difficult organizational challenges."

Alexandra Schmidt

Enterprise UX Playbook

December 1, 2022