Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
JJ Kercher
A Roadmap for Maturing Design in the Enterprise
2018 • Enterprise Experience 2018
Gold
Laura Weiss
There is No Playbook: Leader as Coach During Challenging Times
2024 • Rosenfeld Community
Anna Avrekh
Diversity In and For Design: Building Conscious Diversity in Design and Research
2021 • Design at Scale 2021
Gold
Noreen Whysel
Short Take #4: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Mitchell Bernstein
Organizing Chaos: How IBM is Defining Design Systems with Sketch for an Ever-Changing AI Landscape
2021 • DesignOps Summit 2021
Gold
Wendy Johansson
Design at Scale: Behind the Scenes
2021 • Enterprise Community
Patrick Boehler
The service shift: transforming media organizations to create real value through design
2025 • Advancing Service Design 2025
Gold
Sarah Auslander
Incremental Steps to Drive Radical Innovation in Policy Design
2022 • Civic Design 2022
Gold
Louis Rosenfeld
How AI will Change DesignOps Tooling
2023 • DesignOps Summit 2023
Gold
Sara Logel
Your Colleagues are Your Users Too
2023 • Advancing Research 2023
Gold
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools
2023 • Advancing Research Community
Julie Gitlin
Design as an Agent of Digital Transformation at JPMC
2021 • Design at Scale 2021
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • DesignOps Summit 2021
Gold
Caroline Jarrett
Have fun with statistics?
2024 • Rosenfeld Community
Aras Bilgen
Research Democratization: A Debate
2023 • Advancing Research 2023
Gold
Cheryl Platz
Demystifying Multimodal Design: The Design Practice You Didn't Know You're Doing
2024 • Rosenfeld Community

More Videos

Maria Giudice

"Before you accept a mission as a changemaker, ask yourself do you have a clear directive, champion support, and the right resources?"

Maria Giudice

Becoming a Changemaker by Leading with Design

March 29, 2023

Anat Fintzi

"Empathy isn’t about absorbing everyone’s emotions; it’s about shifting from problem empathy to solution empathy."

Anat Fintzi Rachel Minnicks

Delivering at Scale: Making Traction with Resistant Partners

June 9, 2022

Jaime Creixems

"Iteration is the golden rule of design, including iterating on your design system rules themselves."

Jaime Creixems

Best Practices when Creating and Maintaining a Design System

June 7, 2023

Indi Young

"If you try to describe a person as 'the grumbler' across all their life, you miss the context and purpose."

Indi Young

Thinking styles: Mend hidden cracks in your market

January 8, 2025

Louis Rosenfeld

"Our code of conduct is not window dressing; it’s the front end of a process to keep everyone safe."

Louis Rosenfeld

Welcome / Housekeeping

June 6, 2023

Crystal Philcox

"You have to put on many faces — different skills and mindsets — to communicate with purchasing, finance, legal, and HR."

Crystal Philcox

The Many Faces of Operations

November 6, 2017

Leah Buley

"You can really see how rapidly animation changed in just eight years under Walt Disney’s watch."

Leah Buley

Closing Plenary: The Crisis of Digital

March 31, 2020

Sheryl Cababa

"If you’re working at scale, many problems that seem like 'clock' problems actually have 'cloud' elements."

Sheryl Cababa

Living in the Clouds: Adopting a Systems Thinking Mindset

June 6, 2023

Erika Flowers

"Hope is the only thing that lets the conversation continue and reach forward."

Erika Flowers

AI-Readiness: Preparing NASA for a Data-Driven, Agile Future

June 10, 2025