Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Design at Scale 2022
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Laurent Christoph
Scale the impact of DesignOps in 3D: Diligence, Decision, Discipline
2025 • DesignOps Community
Samuel Proulx
Designing beyond caricatures: Embracing real, diverse user needs
2024 • Advancing Service Design 2024
Gold
Anna Avrekh
User Research, Design, and Product - A Love Story
2021 • Advancing Research 2021
Gold
Louis Rosenfeld
How to use the Rosenbot
2026 • Rosenfeld Community
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Chris Hammond
Embedding sustainability into enterprise design and development: A journey towards "sustainability consciousness"
2025 • Climate UX Interest Group
Josh Clark
Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)
2025 • Rosenfeld Community
Lija Hogan
Doing more with more: Lessons from the Front Lines of Democratization
2022 • Advancing Research 2022
Gold
Jill Fruchter
Inconvenient Insights: The Researcher's Role is to Stay Curious
2023 • Advancing Research 2023
Gold
Crystal Yan
Building a Customer-Centric Culture
2020 • Advancing Research 2020
Gold
Chris Chapo
Data Science and Design: A Tale of Two Tribes
2015 • Enterprise UX 2015
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold
Rob Mitzel
The Tale of Two Companies: Building a Successful UX Practice in a Century-Old Enterprise
2024 • Enterprise Experience 2020
Gold

More Videos

Catt Small

"The more you can help product managers reduce risk, the more they will look at you as their best friend."

Catt Small

Moving from Execution to Strategy as a Designer

December 6, 2022

Laura Schaefer

"Rules and responsibilities documents give people concrete steps they can take towards accessibility."

Laura Schaefer

DesignOps: A Conduit for Inclusion

September 9, 2022

Eric Shumake

"The hardest part is knowing what needs to change but not having the influence to change it."

Eric Shumake

Diagnosis UX: Building Influence in Healthcare Design

April 9, 2026

Marieke McCloskey

"We can’t be there for every behavior after the nudge, nor do we necessarily want to be."

Marieke McCloskey

User Science: Product Analytics & User Research

March 11, 2021

Melissa Eggleston

"In procurement, throwing things over the fence just doesn’t work; collaboration builds trust."

Melissa Eggleston Maya Israni Florence Kasule Owen Seely Andrea Schneider

Practical People Skills for Building Trust on Teams and with Partners

December 9, 2021

Jess Greco

"The business doesn’t just want to do design better; they want the outcomes that design can make possible."

Jess Greco

Creating a Basis for Change: Scaling Design Maturity

June 8, 2022

Teresa Swingler

"We make life or death decisions with design much more often than typical software design."

Teresa Swingler

Look, Up in the Sky! UX/UI for Aerospace

October 27, 2022

"If it’s not usable it’s not valuable."

Product and Design at Bloomberg: A 15-year Evolution

December 6, 2022

Michal Anne Rogondino

"Nearly 30 years into my career, I’m doing the most rewarding work of my life helping modernize military space systems."

Michal Anne Rogondino

Saving Outer Space: The First UX Design System for Our Nation’s Satellites

January 8, 2024