Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters

Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO design team represents just 1% of the IT org yet supports UX for 385,000 employees across the whole company.

  • Over half of IBMers have been with the company less than five years, adding urgency to delivering great onboarding and daily experiences.

  • About 30% of IBM employees work remotely or offsite, increasing reliance on intuitive, well-integrated digital tools.

  • The team follows a mission to lead with design, engineer with excellence, deepen agile practices, and secure the enterprise.

  • Kristen’s team moved from reactive user research to building proactive, predictive capabilities by aggregating decades of experience data.

  • Measurement shifted from isolated metrics like NPS to a holistic scorecard including ease of use, capability, and goal completion.

  • A voice of the employee continuous feedback tool replaced large, infrequent surveys to capture real-time user sentiment.

  • Data-driven decisions led to major redesigns of IBM’s employee directory, help system, and corporate intranet, each achieving significant user appreciation.

  • Measurement can be challenging due to resistance and attempts to game survey responses, underscoring the need for data integrity.

  • Embracing design ops chaos and continuous iteration is key to maturing UX capability in a large enterprise organization like IBM.

Notable Quotes

"This circle represents the size of IBM, 385,000 IBMers. The CIO team is only 3% of that, and our design team is just 1% of the CIO."

"We have about 5,000 new IBMers arriving every month, with high expectations and low tolerance for poor experiences."

"The state of IT is a daily reflection of what the company thinks and feels about its employees."

"You can't just jump to the top of the pyramid like Maslow’s hierarchy—you have to consider all the underlying layers supporting the experience."

"Fletcher Prevend took us to the next level by reporting design and user research directly into the CIO office—a huge differentiator."

"Every minute spent struggling with an IT system is a minute not spent delivering value for IBM and our clients."

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

"We want to send fewer mailers but make them more impactful, using A/B testing to optimize open and click rates."

"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."

"We’re the voice of the employee—we’re arbiters of truth, defenders of experience, and sometimes validators against manipulation."

Ask the Rosenbot
Erin Malone
Understanding the past to prepare for the future
2024 • Rosenfeld Community
Billy Carlson
Principles of Team Wireframing
2023 • DesignOps Summit 2023
Gold
Patrick Boehler
Fishing for Real Needs: Reimagining Journalism Needs with AI
2025 • Designing with AI 2025
Gold
Jaskiran Kang
Why Community is Key to Professionalizing Design
2022 • Civic Design Community
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Alastair Simpson
Debunking the Myths of Cross-Disciplinary Collaboration
2019 • DesignOps Summit 2019
Gold
George Abraham
Design Systems To-Go: Indigo.Design Overview and Exploring the Developer Workflow (Part 3)
2021 • DesignOps Summit 2021
Gold
Eric Shumake
An AMA on UX's Role in Healthcare
2026 • Rosenfeld Community
Dave Malouf
Panel: Design Systems and Documentation
2017 • DesignOps Summit 2017
Gold
Benjamin Wiedmaier
Redefining Toolkits: Unbundling to Create a Perfect Match
2025 • Advancing Research 2025
Gold
Satyam Kantamneni
Do You Have an Experience Vision?
2023 • Enterprise Community
Kritika Sony
Moving AI offscreen: Exploring failures, constraints, and recovery in physical game design
2026 • Designing with AI 2026
Conference
Shan Shen
Translating UX Terms into Business Contexts
2023 • Design in Product 2023
Gold
Margot Bloomstein
Fostering Trust in Your Brand and Beyond
2020 • Enterprise Community
Lada Gorlenko
Theme 1: Intro
2024 • Enterprise Experience 2020
Gold
Peter Van Dijck
Building new AI skills: Creating outsized UX value with evals
2026 • Designing with AI 2026
Conference

More Videos

Daniel J. Rosenberg

"You can’t evaluate bias if the AI can’t explain itself."

Daniel J. Rosenberg

Designing with and for Artificial Intelligence

August 11, 2022

"Design challenges in interviews don’t really help and are often inequitable and exhausting for candidates juggling full-time jobs and family."

DesignOps and The Great Talent War of 2021

August 19, 2021

Liwei Dai

"People don’t always get straight to their question; they might need to do some investigation first before they figure out what to ask."

Liwei Dai

The Heart and Brain of the AI Research

March 31, 2020

Gina Mendolia

"Small feedback comments like, 'Hey, how’s that going?' keep the momentum alive in addition to structured meetings."

Gina Mendolia Jasmine Toy

Coordinated collaboration: a Service Design & DesignOps love story

November 19, 2025

Jon Temple

"Rollout is more successful when management can clearly answer what's in it for the researcher and be honest about trade-offs."

Jon Temple Kalee Dankner Bruce Falk Lauren Galanter

Panel: Stacks, Security, and Stakeholders: The Hidden Work of UXR Tool Procurement

March 12, 2026

Mark Interrante

"If you want to get faster output, optimize the work and workflow first — not add more hours."

Mark Interrante

Collaboration Flows in Product Development

June 9, 2017

Bria Alexander

"There are no overlaps between sponsor content and main programming, so you never have to choose what to attend."

Bria Alexander

Opening Remarks

September 30, 2021

Peter Van Dijck

"Agents are literally models using tools in a loop—they respond to themselves and the tools in chat-like conversations."

Peter Van Dijck

Hands on AI #3: Claude Code for UX people

October 22, 2025

Rachael Dietkus, LCSW

"Assume everybody has the potential for trauma to show up in any interaction you have with them."

Rachael Dietkus, LCSW Uday Gajendar Dr. Dawn Emerick Dawn E. Shedrick, LCSW

Leading through the long tail of trauma

July 7, 2022