Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters

Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO design team represents just 1% of the IT org yet supports UX for 385,000 employees across the whole company.

  • Over half of IBMers have been with the company less than five years, adding urgency to delivering great onboarding and daily experiences.

  • About 30% of IBM employees work remotely or offsite, increasing reliance on intuitive, well-integrated digital tools.

  • The team follows a mission to lead with design, engineer with excellence, deepen agile practices, and secure the enterprise.

  • Kristen’s team moved from reactive user research to building proactive, predictive capabilities by aggregating decades of experience data.

  • Measurement shifted from isolated metrics like NPS to a holistic scorecard including ease of use, capability, and goal completion.

  • A voice of the employee continuous feedback tool replaced large, infrequent surveys to capture real-time user sentiment.

  • Data-driven decisions led to major redesigns of IBM’s employee directory, help system, and corporate intranet, each achieving significant user appreciation.

  • Measurement can be challenging due to resistance and attempts to game survey responses, underscoring the need for data integrity.

  • Embracing design ops chaos and continuous iteration is key to maturing UX capability in a large enterprise organization like IBM.

Notable Quotes

"This circle represents the size of IBM, 385,000 IBMers. The CIO team is only 3% of that, and our design team is just 1% of the CIO."

"We have about 5,000 new IBMers arriving every month, with high expectations and low tolerance for poor experiences."

"The state of IT is a daily reflection of what the company thinks and feels about its employees."

"You can't just jump to the top of the pyramid like Maslow’s hierarchy—you have to consider all the underlying layers supporting the experience."

"Fletcher Prevend took us to the next level by reporting design and user research directly into the CIO office—a huge differentiator."

"Every minute spent struggling with an IT system is a minute not spent delivering value for IBM and our clients."

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

"We want to send fewer mailers but make them more impactful, using A/B testing to optimize open and click rates."

"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."

"We’re the voice of the employee—we’re arbiters of truth, defenders of experience, and sometimes validators against manipulation."

Ask the Rosenbot
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Robin Beers
Navigating organizational systems: Rethinking researcher’s role in driving change
2025 • Advancing Research 2025
Gold
Nick Lewis
Designing and building low-carbon websites independently
2025 • Climate UX Interest Group
Rachel Posman
A Closer Look at Team Ops and Product Ops (Two Sides of the DesignOps Coin)
2020 • DesignOps Community
Uday Gajendar
Leading through the long tail of trauma
2022 • Advancing Research Community
Aaron Stienstra
Leveraging Civic Design to Advance Equity and Rebuild Trust in the US Federal Government
2021 • Civic Design 2021
Gold
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Noz Urbina
Rapid AI-powered UX (RAUX): A framework for empowering human designers
2025 • Rosenfeld Community
Melinda Belcher
Insider preview of Enterprise Experience 2020
2020 • Enterprise Community
Jemma Ahmed
Theme 2 Intro
2024 • Advancing Research 2024
Gold
Jim Kalbach
Peace is waged with sticky notes: Mapping Real-World Experiences
2018 • Enterprise Experience 2018
Gold
Eduardo Ortiz
Day 3 Theme Panel
2025 • Advancing Research 2025
Gold
Caroline Jarrett
Have fun with statistics?
2024 • Rosenfeld Community
Marissa Cui
Climate Design Product Showcase
2024 • Climate UX Interest Group
Peter Merholz
Customer-Centered Design Organizations
2017 • Enterprise Experience 2017
Gold
Adam Cutler
Discussion
2016 • Enterprise UX 2016
Gold

More Videos

Mariesa Lenz

"High functioning teams are built on trust and not control."

Mariesa Lenz

What Beekeeping Taught me about Product Teams

October 29, 2025

Jorge Arango

"Large language models don’t think like us but have superpowers in pattern matching that help surface relationships in linked notes."

Jorge Arango

Exploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars)

April 19, 2024

Peter Van Dijck

"Skills are like onboarding material for the model; you write them as simple markdown files that describe workflows or tools."

Peter Van Dijck

Hands on AI #3: Claude Code for UX people

October 22, 2025

Ovetta Sampson

"Good UX and good design need not only to make a comeback but lead the way in this era of AI."

Ovetta Sampson

Managing the Human Engagement Risks of AI

June 10, 2025

B. Pagels-Minor

"My mom is four foot eleven but looked like a six foot six linebacker when she stood up for me."

B. Pagels-Minor

Breaking the Tension: The Power of Enabling Your Employees to Show Up Authentically

June 10, 2022

Amy Bucher

"Nudges are not a magic wand; they can help but need to fit within a larger system and behavioral pathway to truly work."

Amy Bucher

Harnessing behavioral science to uncover deeper truths

March 12, 2025

Matt Bernius

"Trauma-informed is not a destination; it’s a journey, a continuous state of becoming."

Matt Bernius Sarah Fathallah Hera Hussain Jessica Zéroual-Kara

Trauma-informed Research: A Panel Discussion

October 7, 2021

Sam Proulx

"Customizability is about meeting people where they are, whether that’s on mobile, desktop, voice interaction, or different communication channels."

Sam Proulx

Online Shopping: Designing an Accessible Experience

November 29, 2023

Kritika Sony

"You don’t have to build a game; maybe you want to build a toy for your cat or a lamp—it just has to be yours."

Kritika Sony

Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design

June 10, 2026