Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Making it Count: Developing a custom digital metric framework that works
Friday, October 15, 2021 • QuantQual Interest Group
Share the love for this talk
Making it Count: Developing a custom digital metric framework that works
Speakers: Alberto Ferreira
Link:

Summary

NPS, SUS, HEART, CSAT, CES, CLI—the list of ambiguous acronyms goes on and on. Companies are under more pressure than ever to measure and quantify their results and their interactions with customers, but finding the right metric and the right approach is a challenging process that risks leaving key factors behind as you commit to one sole standard. However, what if there is another way to measure digital transformation and how people see your services? This is what this presentation will focus on. You will learn how to navigate the pitfalls of standardized metrics—with their pros and cons—and learn how to build and implement a custom metric framework that incorporates the best aspects of Net Promoter Score, Customer Effort Score, System Usability Scale, and other, into one cohesive and modern whole aimed at developing the actionability and traceability of your operations and customer services. Attendees takeaways include: how to develop a custom framework without losing benchmarking capability, identifying gaps and needs from this framework that is not focused solely on operational numbers, and how to wade through the murky waters of digital experience quantification. Bring your best questions and leave with actionable insights that you can put in use immediately.

Key Insights

  • NPS often obscures diverse customer experiences by aggregating different user segments into a single score.

  • Custom metric frameworks combining behavioral, attitudinal, and quality-of-experience data yield richer insights than standardized metrics.

  • Separating attitudinal (feelings) and behavioral (actions) metrics avoids the risk of conflating different dimensions of user experience.

  • Quantitative data should be supplemented with qualitative verbatims analyzed via sentiment analysis for fuller context.

  • Business leadership prefers simple headline metrics, so complex multi-dimensional data must be distilled carefully, often accompanied by storytelling.

  • Existing UX metrics frameworks like NASA TLX, SUS, SuperQ, and Google HEART offer valuable components but may need adaptation for specific digital contexts.

  • Designing useful metrics starts by clarifying goals, then defining quantity, signals, timeframe, and scope explicitly.

  • High conversion rates alone can be misleading without considering total volume of conversions and user behaviors in context.

  • Customer sentiment is complex and may not align directly with retention or recommendation behavior, so metrics must capture nuance.

  • Short, targeted surveys on key user journeys help balance depth of insight with user willingness to respond.

Notable Quotes

"A user-centered measurement should not be one dimensional because people are complex."

"NPS scores can rise overall, yet new and existing customers might have completely different views that get obscured."

"A metric is usually the result of a relation between two measures, not interchangeable with a measure itself."

"Customers tend to inflate negative feelings in surveys, but that doesn't always reflect their actual behavior."

"We aim to keep the reporting simple for stakeholders while incorporating multiple data points in the background."

"Combining behavioral and attitudinal data helps us understand not just what users do, but how they feel about it."

"We use sentiment analysis on verbatims to quantify positive, neutral, and negative customer feedback over time."

"Designing a metric starts with the goals, not the metric itself."

"Conversion rate alone isn’t sufficient; total conversions and traffic volume also matter."

"Including human stories alongside numbers helps business leaders grasp the complexity behind the metrics."

Ask the Rosenbot
Jemma Ahmed
Convergent Research Techniques in Customer Journey Mapping
2020 • Advancing Research 2020
Gold
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Prayag Narula
How to Empower Your Designers to Do Good Research – And Why You Want To
2022 • Design at Scale 2022
Gold
Dave Gray
Liminal Thinking: Sense-making for systems in large organizations
2015 • Enterprise UX 2015
Gold
Sara Asche Anderson
Not Your Ordinary Re-Brand: Design's Path to Driving Customer Obsession at Best Buy
2024 • Enterprise Experience 2020
Gold
Alexandra Schmidt
Why Ethics Can't Save Tech
2022 • Civic Design 2022
Gold
Sarah Flamion
Complex Problem? Add Clarity by Combining Research and Systems Thinking
2020 • Advancing Research 2020
Gold
Brendan Jarvis
It was the Best of Times. It was the Worst of Times.
2024 • DesignOps Summit 2024
Gold
Erin May
Distributed, Democratized, Decentralized: Finding a Research Model to Support Your Org
2022 • Advancing Research 2022
Gold
Neema Mahdavi
Operationalizing DesignOps
2018 • DesignOps Summit 2018
Gold
Erin Malone
Understanding the past to prepare for the future
2024 • Rosenfeld Community
Tricia Wang
From Users to Shapers of AI: The Future of Research
2024 • Advancing Research 2024
Gold
Erin Hauber
Design is Not the Frosting on the Scaled Agile Layer Cake
2019 • DesignOps Summit 2019
Gold
Luke Roberts
Panel Discussion
2024 • Advancing Service Design 2024
Gold
Jackie Ajoux
Leveling-Up: A Single-Player’s Guide to the DesignOps Team-of-One
2024 • DesignOps Summit 2020
Gold
Francesca Barrientos, PhD
You Need Your Own Definition of Design Maturity
2022 • Design at Scale 2022
Gold

More Videos

Robin Beers

"We’re not going back to stability; we’re riding this wave of perpetual upheaval and must learn to surf it."

Robin Beers

Navigating organizational systems: Rethinking researcher’s role in driving change

March 13, 2025

Laura Schaefer

"Putting accessibility and inclusion in writing makes it real and formal, not just a spoken idea."

Laura Schaefer

DesignOps: A Conduit for Inclusion

September 9, 2022

Ned Gartside

"The digital sector’s compounded impact on carbon footprint exceeds industries like airlines when aggregated across millions."

Ned Gartside Mike Gifford Zoe Lopez-Latorre Tzviya Siegman

Navigating accessibility and climate

April 17, 2024

Peter Van Dijck

"You should get good at prompt engineering by using AI a lot, not by memorizing outdated tricks."

Peter Van Dijck

Designing AI-first products on top of a rapidly evolving technology

June 10, 2025

Surya Vanka

"Visual process maps embody cognition and allow people to step through design without becoming experts."

Surya Vanka

Unleashing Swarm Creativity to Solve Enterprise Challenges

June 10, 2021

Jemma Ahmed

"Insight practices are just getting more established and are asking questions of higher strategic value, bringing disciplines closer together."

Jemma Ahmed

Convergent Research Techniques in Customer Journey Mapping

March 31, 2020

"Some companies have gotten ahead by making 100% remote work an official policy, which is a big advantage in hiring."

DesignOps and The Great Talent War of 2021

August 19, 2021

Robin Beers

"The executive’s interpretation was completely out of line with the customers’ realities."

Robin Beers

Beyond Insights: Researchers as Organizational Change Catalysts

March 25, 2024

Maria Skaaden

"We keep our playbook in a place where anyone can edit it from the team."

Maria Skaaden

Panel Discussion: Methodologies and Work Environments

November 8, 2018