Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works

Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Louis Rosenfeld
Day 1 Welcome
2024 • DesignOps Summit 2024
Gold
Johnny Michaelsen
Measure Behaviors, Not Results
2026 • Rosenfeld Community
Jen Briselli
Learning Is The Engine: Designing & Adapting in a World We Can’t Predict
2025 • Rosenfeld Community
Frances Yllana
The Big Question about Impact: A Panel Discussion
2024 • DesignOps Summit 2024
Gold
Sean McKay
Coexisting with non-researchers: Practical strategies for a democratized research future
2025 • Advancing Research 2025
Gold
Meaghan Waters
Lack of Product Thinking will Doom Your Legacy Modernization
2021 • Design at Scale 2021
Gold
Monty Hammontree
The Future of UX Research
2020 • Advancing Research Community
Rachael Dietkus, LCSW
Everything You Need to Know about the Civic Design 2022 Call for Presentations
2022 • Civic Design Community
Marissa Cui
Climate Design Product Showcase
2024 • Climate UX Interest Group
Anna Poznyakov
Get The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact
2021 • Advancing Research 2021
Gold
David Eisner
From prototype to production: Vibe coding design for real engineering systems
2026 • Designing with AI 2026
Conference
Lukas Moro
“Feels Like Paper!”: Interfacing AI through Paper
2025 • Designing with AI 2025
Gold
Brad Peters
Short Take #1: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Matt Duignan
HITS, Microsoft's internal human insight system: From research library to living body of knowledge
2019 • Advancing Research Community
Natalie Dunbar
DesignOps and Content Strategy: Envisioning the Future Together
2021 • DesignOps Summit 2021
Gold
Kate Towsey
Participant Recruitment and Management Tools
2026 • Advancing Research 2026
Gold

More Videos

Frances Yllana

"We don't do PRDs anymore; we just vibe code, but that still means putting the right prompt behind it."

Frances Yllana Kaaren Hanson Husani Oakley Dan Olsen

DesignOps Exposed: What do our peers really think of us?

September 11, 2025

Nathan Shedroff

"Critical making requires critique, theory, activity, reflection, leading back to theory, but it doesn't have to start with theory. It can start anywhere."

Nathan Shedroff Hugh Dubberly Thomas J. McLeish

How Will Design be Taught When the Schools Shut Down?

May 8, 2026

Spencer L. A. Stultz

"Intersectionality is a lens to understand how dominant culture creates obstacles outside of conventional advocacy."

Spencer L. A. Stultz

Why Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives

October 4, 2023

Louis Rosenfeld

"MJ Broadband’s sketch notes will make your conference experience 10 times easier."

Louis Rosenfeld Bria Alexander

Opening Remarks

November 16, 2022

Bria Alexander

"If you want to take part in the gifs, laughs, jokes, questions, and links to resources, join us on #deogenerall Slack channel."

Bria Alexander

Opening Remarks

September 30, 2021

Kevin Bethune

"Servant leadership means divvying up the work and giving team members runway to lead parts of the vision."

Kevin Bethune

Reimagining Design: Unlocking Strategic Innovation

June 8, 2022

Kim Lenox

"Sharing meals is a really critical part of building trust which seems kind of strange but breaking bread together makes a difference."

Kim Lenox

Leading Distributed Global Teams

May 20, 2019

Erika Flowers

"Designers must learn code fundamentals to use AI effectively, akin to becoming apprentices to the world’s greatest software developers."

Erika Flowers

The Handoff is Dead: Design-Led Engineering with AI Agents

March 4, 2026

Alla Weinberg

"Mental health challenges are now the norm among employees across all organizational roles."

Alla Weinberg

Workers Are Sick of Change: The Cure is Psychological Safety

June 6, 2023