Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Sarah Alvarado
How to make UX research leadership more effective [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Jemma Ahmed
Theme 2 Intro
2022 • Advancing Research 2022
Gold
Amy Thibodeau
Opening Keynote: Process and Ambiguity
2019 • DesignOps Summit 2019
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Rittika Basu
Age and Interfaces: Equipping Older Adults with Technological Tools
2023 • Advancing Research Community
Ovetta Sampson
Turning UX Passion into Real Product Influence
2023 • Enterprise UX 2023
Gold
Crystal Philcox
The Many Faces of Operations
2017 • DesignOps Summit 2017
Gold
Bria Alexander
Welcome
2022 • DesignOps Summit 2022
Gold
Tim Frick
The journey of building a sustainable design practice
2025 • Climate UX Interest Group
Deanna Washington
Scaling Success: Paving the Path from DesignOps to VP
2023 • DesignOps Summit 2023
Gold
Holly Cole
Understanding Experiences: When you have to do more than work
2018 • DesignOps Summit 2018
Gold
Lisa Spitz
Building Trust Through Equitable Research Practices
2022 • Civic Design 2022
Gold
Russ Unger
Onboarding: The Ecosystem, not the Afterthought
2017 • DesignOps Summit 2017
Gold
Mark Templeton
Creating a Legacy: the ultimate experience
2017 • Enterprise Experience 2017
Gold
Anna Nguyen
Why Our Voice of the Customer is Better Than Yours
2022 • Advancing Research 2022
Gold
Mariah Hay
BUILD: Discussion
2018 • Enterprise Experience 2018
Gold

More Videos

Jamie Beck Alexander

"Quantities can be measured, but qualities need to be mapped — that shift feels core to the designer's role."

Jamie Beck Alexander Nina Gregg Shawn Petersen Bill DeRouchey

How can you find your role in climate?

January 17, 2024

Maria Skaaden

"Positive deviants operate on the fringes and get the best results by doing things differently."

Maria Skaaden

Panel Discussion: Methodologies and Work Environments

November 8, 2018

Abbey Smalley

"I got a no. The message I was hearing was, scale the best you can with what you have and make it work."

Abbey Smalley Sylas Souza

Scaling UX Past the Size of Your Team

January 8, 2024

Ariel Kennan

"We’ll hear about creating healthy cultures and ways of working across many organizational silos, including vendors."

Ariel Kennan

Theme Two Intro

November 17, 2022

Megan Kierstead

"The only person who gets to decide what it means to be badass is you."

Megan Kierstead

You Are a Badass at UX: Overcoming Imposter Syndrome

March 10, 2021

Nathan Shedroff

"To be business and design, not business versus design, we have to change the language we use to talk about value."

Nathan Shedroff

Redefining Value: Bridging the Innovation Culture Divide

May 14, 2015

Kristin Sundermeyer

"Some designers forget to stop the Toggl timer, which skews data but is easy to spot and correct."

Kristin Sundermeyer Tygre Morehart

Design Ops Metrics

September 30, 2021

Chui Chui Tan

"In Kenya, flexible attitudes toward time caused delays in sessions, which stressed our team but were culturally normal."

Chui Chui Tan

Global insights: Embracing international and intercultural research for innovation

March 12, 2025

Daniel J. Rosenberg

"Digital medicine apps are legally considered a pharmaceutical, available by prescription only."

Daniel J. Rosenberg

Digital Medicine Design

September 26, 2019