Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Megan Blocker
Panel: Excellence in Impact
2024 • Advancing Research 2024
Gold
Joshua Graves
We Need To Talk: Navigating Conversations with Your Boss (Part 1 of 3)
2025 • Rosenfeld Community
Dalia El-Shimy
So You've Got a Seat at the Table. Now What?
2020 • Advancing Research 2020
Gold
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
Laura Smith
Embedding Service Design and Agile Practice within UK Planning Teams to Create Services that Last
2024 • Advancing Service Design 2024
Gold
Bria Alexander
Opening Remarks
2023 • Advancing Research 2023
Gold
Kristin Skinner
Five Years of DesignOps
2021 • DesignOps Summit 2021
Gold
Lada Gorlenko
Theme 2 Intro
2022 • Design at Scale 2022
Gold
World Usability Day Panel Discussion
2022 • DesignOps Community
Product and Design at Bloomberg: A 15-year Evolution
2022 • Design in Product 2022
Gold
Taylor Klassman
Shaping the Next Era of UX Research: Collaborative Forum
2025 • Advancing Research 2025
Gold
Amy Thibodeau
Opening Keynote: Process and Ambiguity
2019 • DesignOps Summit 2019
Gold
Ian Johnson
Latent Scope: Finding structure in unstructured data
2025 • Designing with AI 2025
Gold
Samuel Proulx
From Standards to Innovation: Why Inclusive Design Wins
2025 • DesignOps Summit 2025
Conference
Dan Willis
Enterprise Storytelling Sessions
2016 • Enterprise UX 2016
Gold
Michele Wong
Helping Them Help Us
2024 • Enterprise Experience 2020
Gold

More Videos

Jon Fukuda

"We encourage you all to be your own heroes and connect your own documents."

Jon Fukuda Ellie Krysl

Design Planning and Management Support

October 3, 2023

Saara Kamppari-Miller

"Who works here anyways? I knew about immediate teams but beyond that, it was a fog."

Saara Kamppari-Miller

Cartography for Design Communities

September 10, 2025

Kevin Bethune

"Servant leadership means divvying up the work and giving team members runway to lead parts of the vision."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Daniel Orbach

"It took over a year and 10 teams just to ship four onboarding screens."

Daniel Orbach

Zero to One: Co-Creating Operating Models with your Team

September 23, 2024

Marc Fonteijn

"If you don’t feel connected and you’re standing alone, the challenge looks much bigger."

Marc Fonteijn Ru Butler

Increase your confidence, influence, and impact (through a Professional Community)

December 3, 2024

Clara Kliman-Silver

"It’s critical to measure perceived value and ask: Are you really solving the right problem with AI?"

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Robin Beers

"Synthesis is a researcher’s superpower, not just analysis or breaking things down."

Robin Beers Nalini Kotamraju Andy Warr

Panel: Excellence in Communicating Insights

March 26, 2024

Ana Maria Montero Barrantes

"I am not throwing away your UX."

Ana Maria Montero Barrantes Aditi Dhar Michelle Kaplan Nate Osborne Matt Laurence

The Authentic UX Talent Show

January 8, 2024

Kristen Honey

"The peer review process is slow and gatekeeps change, so innovating from the margins is strategic."

Kristen Honey

"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design

November 18, 2021