Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Davis Neable
How to Drive a Design Project When you Don’t Have a Design Team
2021 • Design at Scale 2021
Gold
Benjamin Wiedmaier
Redefining Toolkits: Unbundling to Create a Perfect Match
2025 • Advancing Research 2025
Gold
Megan Blocker
Theme 2 Intro
2025 • Advancing Research 2025
Gold
Dave Malouf
The Past, Present, and Future of DesignOps: a 2-part DesignOps Community Call (Part 2)
2022 • DesignOps Community
Nathan Shedroff
Redefining Value: Bridging the Innovation Culture Divide
2015 • Enterprise UX 2015
Gold
Caroline Vize
The State of UX: Five Lessons from 2021 to Accelerate Digital Experience in 2022
2022 • Advancing Research 2022
Gold
Bria Alexander
Opening Remarks
2023 • DesignOps Summit 2023
Gold
Nicole Aleong
What UX research can learn from other research practices [Advancing Research Community Workshop Series]
2023 • Advancing Research Community
Catherine Courage
The Enterprise UX Journey: Lessons From the Voyage & The Opportunity Ahead
2015 • Enterprise UX 2015
Gold
Steve Baty
Discussion
2016 • Enterprise UX 2016
Gold
Dr. Karl Jeffries
The Science of Creativity for DesignOps
2024 • DesignOps Summit 2020
Gold
Dave Hora
A Research Skills Evolution
2021 • Advancing Research 2021
Gold
Sahibzada Mayed
Cultivating Design Ecologies of Care, Community, and Collaboration
2023 • DesignOps Summit 2023
Gold
Angelos Arnis
Navigating the Rapid Shifts in Tech's Turbulent Terrain
2023 • DesignOps Summit 2023
Gold
Zen Ren
Taking Inspiration from Instructional Design for Research
2022 • Advancing Research 2022
Gold
Jennifer Kong
Journeying toward AI-assisted documentation in healthcare
2024 • Designing with AI 2024
Gold

More Videos

Marissa Cui

"Users want to feel that their dollars are making a difference and want clear storytelling about their impact."

Marissa Cui Rachel He Michael Leggett Manos Saratsis

Climate Design Product Showcase

March 13, 2024

Bria Alexander

"If you feel uncomfortable or bullied, it is so important that you know you can reach out for help."

Bria Alexander

Opening Remarks

October 4, 2023

Lada Gorlenko

"Simply asking stakeholders what success looks like for them in a project is a powerful way to start meaningful conversations."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Carol Scott

"Trauma-informed approaches are like seat belts and airbags for tech products."

Carol Scott Melissa Eggleston

Avoid Harming Your Team and Users: Promoting Care and Brand Reputation with Trauma-Informed UX Practices

February 5, 2025

Ned Dwyer

"AI will soon enable us to automatically flag leading questions and coach researchers for quality."

Ned Dwyer

Right horses for the right courses – how and when to democratize research

November 20, 2025

Anna Nguyen

"We don’t just want to make customers satisfied, but provide an amazing experience and get those around us excited about what we’re doing."

Anna Nguyen Emily Brogan

Why Our Voice of the Customer is Better Than Yours

March 10, 2022

Nicole Aleong

"Our deepest fear is not that we’re inadequate, but that we’re powerful beyond measure."

Nicole Aleong Michaela Mora Prayag Narula Brianna Sylver

What UX research can learn from other research practices [Advancing Research Community Workshop Series]

September 14, 2023

Brianna Sylver

"To elevate design, you have to upskill in business acumen and one or more technical or specialty disciplines."

Brianna Sylver

Lead With Purpose

March 31, 2020

Josh Clark

"We can’t design for every possible outcome anymore. We have to anticipate fuzzy ranges of results and channel behavior accordingly."

Josh Clark Veronika Kindred

Sentient Design, AI, and the Radically Adaptive Experience (1st of 3 seminars)

January 15, 2025