Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Kristin Sundermeyer
Design Ops Metrics
2021 • DesignOps Summit 2021
Gold
Brendan Jarvis
Framing Tomorrow by Questioning Today
2022 • Design at Scale 2022
Gold
Tony Turner
Capturing Deep Insights
2021 • DesignOps Summit 2021
Gold
Amy Parness
Scaling Sustainability: Complementary strategies that drive long-term success
2025 • Climate UX Interest Group
Jemma Ahmed
Theme 2 Intro
2024 • Advancing Research 2024
Gold
Erin Hoffman-John
This Game is Never Done: Design Leadership Techniques from the Video Game World
2017 • DesignOps Summit 2017
Gold
Adam Thomas
Survival Metrics – Making Change in a Fast, Data-Informed, and Politically Safe Way
2022 • Design in Product 2022
Gold
Ellie Krysl
Planned Right. Managed Right. Designed Right.
2023 • Enterprise UX 2023
Gold
Trisha Terhar
Empathizing with the Empowered: Non-Researcher Responses to Democratization
2022 • Advancing Research 2022
Gold
Greg Petroff
Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
2025 • Rosenfeld Community
Jane Reid
Self-care in User Research
2020 • Advancing Research Community
Jason Mesut
Shaping design, designers and teams
2018 • DesignOps Summit 2018
Gold
Lori Muszynski
Keeping Design Weird
2023 • DesignOps Summit 2023
Gold
Laura Schaefer
DesignOps: A Conduit for Inclusion
2022 • DesignOps Summit 2022
Gold
Toby Haug
Discussion
2017 • Enterprise Experience 2017
Gold
Bria Alexander
Opening Remarks
2023 • DesignOps Summit 2023
Gold

More Videos

Dominique Ward

"Conway’s law means the way we communicate replicates our organizational structure, creating silos we must resist."

Dominique Ward

The Most Exciting Time for DesignOps is Now

September 8, 2022

Sam Proulx

"If you want an accessible app on iOS and Android, using the standard native APIs and controls will get you there."

Sam Proulx

Mobile Accessibility and You

June 9, 2022

Kristin Sundermeyer

"The yellow 80% allocation guide reminds that designers shouldn't be expected at full 100% project time due to meetings and breaks."

Kristin Sundermeyer Tygre Morehart

Design Ops Metrics

September 30, 2021

Daniel Gloyd

"The Shaker village sidewalks followed people’s natural inclinations, not forcing unnatural movement."

Daniel Gloyd

Warming the User Experience: Lessons from America's first and most radical human-centered designers

May 9, 2024

Brenna Fallon

"If you get a perfect OKR score, it means you didn’t set your sights high enough."

Brenna Fallon

Learning Over Outcomes

October 24, 2019

Ian Johnson

"We found patterns that product teams didn’t expect or even want to look for."

Ian Johnson

Latent Scope: Finding structure in unstructured data

June 11, 2025

Jason Mesut

"Directive Disruptors plow through organizations without listening, causing alienation and low morale."

Jason Mesut Martina Hodges-Schell Jose Coronado

Unmasking Design Leadership: Navigating leadership without neglecting ourselves

October 30, 2025

Dean Broadley

"Not belonging anywhere meant that I had access everywhere."

Dean Broadley

Not Black Enough to be White

January 8, 2024

John Calhoun

"Your next design ops peak might actually be beyond the walls of your current design practice."

John Calhoun

Have we Reached Our Peak? Spotting the Next Mountain For DesignOps to Climb

October 1, 2021