Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Erin Weigel
Get Your Whole Team Testing to Design for Impact
2024 • Rosenfeld Community
Jen Cardello
Learning Velocity—The Insights Speedometer
2021 • Advancing Research Community
Soma Ghosh
What emerging methods are advancing UX research [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Susan Weinschenk
Evaluating the Maturity of UX in Your Organization
2020 • Enterprise Community
Chris Geison
What's Next for Research?
2021 • Advancing Research Community
Andrea Gallagher
The Problem Space
2019 • Advancing Research Community
Brigette Metzler
Scaling ResearchOps: Helping Researchers do Their Best Work
2020 • Advancing Research 2020
Gold
Yalenka Mariën
Designing for Digital Inclusion in the Belgian Government
2021 • Civic Design 2021
Gold
Michael Land
Establishing Design Operations in Government
2021 • DesignOps Community
Erika Flowers
AI-Readiness: Preparing NASA for a Data-Driven, Agile Future
2025 • Designing with AI 2025
Gold
Amelia Cole
Data-Prompted Interviews
2021 • QuantQual Interest Group
Russ Unger
Onboarding: The Ecosystem, not the Afterthought
2017 • DesignOps Summit 2017
Gold
Jemma Ahmed
Theme 2 Intro
2022 • Advancing Research 2022
Gold
Sara Conklin
A UXer’s 12-Month Journey from Climate Concern to Climate Credibility
2025 • Climate UX Interest Group
Prayag Narula
HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer
2023 • Advancing Research 2023
Gold
Marc Fonteijn
Increase your confidence, influence, and impact (through a Professional Community)
2024 • Advancing Service Design 2024
Gold

More Videos

Yoel Sumitro

"Students cannot be taught what they need to know; they can only be coached to absorb it through demonstration."

Yoel Sumitro

Actions and Reflections: Bridging the Skills Gap among Researchers

March 9, 2022

Zariah Cameron

"Share everything, own nothing but credit everyone."

Zariah Cameron

ReDesigning Wellbeing for Equitable Care in the Workplace

September 23, 2024

Doug Powell

"Disruptive change is the new normal, coming from every direction — not just technology."

Doug Powell

DesignOps and the Next Frontier: Leading Through Unpredictable Change

September 11, 2025

Darian Davis

"A common toxic behavior is glory seeking, like presenting work as your own when it was a team effort."

Darian Davis

Lessons from a Toxic Work Relationship

January 8, 2024

Mike Oren

"I helped a company save 12 million dollars by not investing in an iris scanner in a one-week project."

Mike Oren

Why Pharmaceutical's Research Model Should Replace Design Thinking

March 28, 2023

Jacqui Frey

"One operations person for every ten designers is when you start to feel the pain of coordination."

Jacqui Frey

Scale is Social Work

March 19, 2020

Kim Holt

"The V2MOM stands for Vision, Values, Methods, Obstacles, and Measures and is a framework every Salesforce employee uses."

Kim Holt Emma Wylds Pearl Koppenhaver Maisee Xiong

A Salesforce Panel Discussion on Values-Driven DesignOps

September 8, 2022

Samuel Proulx

"You own the design system, which gives you the unique ability to integrate that accessibility thinking into all of your components."

Samuel Proulx

From Standards to Innovation: Why Inclusive Design Wins

September 10, 2025

Laine Riley Prokay

"For early career UXers, compassion could be as simple as expressing gratitude for help."

Laine Riley Prokay

How DesignOps can Drive Inclusive Career Ladders for All

September 30, 2021