Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Measuring What Matters
Gold
Wednesday, October 23, 2019 • DesignOps Summit 2019
Share the love for this talk
Measuring What Matters
Speakers: Kristin Wisnewski
Link:

Summary

Research and analytics are the eyes and ears of DesignOps. Validating whether strategies are effective or ineffective allows course correction and adaptation. Identifying problems that require action is essential for facilitating collaboration and ensuring that team members are engaged. Prioritizing user needs is the heart of user experience and Agile development. None of this can be done well without an effective research program. We’ll share how we scale research efforts to collect feedback from thousands of users per month and track the impact of dozens of projects involving 120+ designers.

Key Insights

  • IBM’s CIO design team represents just 1% of the IT org yet supports UX for 385,000 employees across the whole company.

  • Over half of IBMers have been with the company less than five years, adding urgency to delivering great onboarding and daily experiences.

  • About 30% of IBM employees work remotely or offsite, increasing reliance on intuitive, well-integrated digital tools.

  • The team follows a mission to lead with design, engineer with excellence, deepen agile practices, and secure the enterprise.

  • Kristen’s team moved from reactive user research to building proactive, predictive capabilities by aggregating decades of experience data.

  • Measurement shifted from isolated metrics like NPS to a holistic scorecard including ease of use, capability, and goal completion.

  • A voice of the employee continuous feedback tool replaced large, infrequent surveys to capture real-time user sentiment.

  • Data-driven decisions led to major redesigns of IBM’s employee directory, help system, and corporate intranet, each achieving significant user appreciation.

  • Measurement can be challenging due to resistance and attempts to game survey responses, underscoring the need for data integrity.

  • Embracing design ops chaos and continuous iteration is key to maturing UX capability in a large enterprise organization like IBM.

Notable Quotes

"This circle represents the size of IBM, 385,000 IBMers. The CIO team is only 3% of that, and our design team is just 1% of the CIO."

"We have about 5,000 new IBMers arriving every month, with high expectations and low tolerance for poor experiences."

"The state of IT is a daily reflection of what the company thinks and feels about its employees."

"You can't just jump to the top of the pyramid like Maslow’s hierarchy—you have to consider all the underlying layers supporting the experience."

"Fletcher Prevend took us to the next level by reporting design and user research directly into the CIO office—a huge differentiator."

"Every minute spent struggling with an IT system is a minute not spent delivering value for IBM and our clients."

"Before, we never really looked holistically at decades of UX data—now we use it to be proactive and avoid duplicative studies."

"We want to send fewer mailers but make them more impactful, using A/B testing to optimize open and click rates."

"Don’t try to control the data to fit your narrative—the goal is to let insights guide your decisions, not justify them."

"We’re the voice of the employee—we’re arbiters of truth, defenders of experience, and sometimes validators against manipulation."

Ask the Rosenbot
Tutti Taygerly
Videconference: How to Work with Difficult People with Tutti Taygerly
2020 • Enterprise Community
Savina Hawkins
Harnessing AI in UXR: Practical Strategies for Positive Impact
2024 • Advancing Research 2024
Gold
Marc Fonteijn
Increase your confidence, influence, and impact (through a Professional Community)
2024 • Advancing Service Design 2024
Gold
Sam Proulx
Prototype Reviews, People With Disabilities, and You
2021 • Civic Design 2021
Gold
Etienne Fang
Power of Insights: Why sharing is better than silos with Uber’s Insights Platform
2019 • Advancing Research Community
Ovetta Sampson
Turning UX Passion into Real Product Influence
2023 • Enterprise UX 2023
Gold
Melissa Eggleston
Practical People Skills for Building Trust on Teams and with Partners
2021 • Civic Design 2021
Gold
Bria Alexander
Opening Remarks
2023 • DesignOps Summit 2023
Gold
Rebecca Gimenez
Work in Progress: Service Design at Airbnb
2024 • Advancing Service Design 2024
Gold
Kate Towsey
Ask Me Anything (AMA) with Kate Towsey
2025 • Rosenfeld Community
Saara Kamppari-Miller
Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
2024 • DesignOps Community
Greg Petroff
Exit Interview #1: Greg Petroff: From Silicon Valley Executive to Sonoma County Possibilitarian
2025 • Rosenfeld Community
Daniel Gloyd
Designing Warmth
2025 • Rosenfeld Community
Neil Barrie
Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
2024 • Advancing Research 2024
Gold
Mac Smith
Measuring Up: Using Product Research for Organizational Impact
2021 • Advancing Research 2021
Gold
Matt Bernius
Trauma-informed Research: A Panel Discussion
2021 • Advancing Research Community

More Videos

Ned Dwyer

"It’s not always a charismatic repository that people flock to, but tools that seek you out where you work are more effective."

Ned Dwyer Emily Stewart James Wallis

The Intersection of Design and ResearchOps

September 24, 2024

Michael Land

"Hiring specs for federal designers don’t exist as we expect—jobs are often classified as public affairs or other unrelated titles."

Michael Land

Establishing Design Operations in Government

February 18, 2021

Sam Proulx

"Blockchain apps aren’t located in one country or jurisdiction, making legal enforcement of accessibility laws difficult."

Sam Proulx

To Boldly Go: The New Frontiers of Accessibility

November 18, 2022

Vasileios Xanthopoulos

"We run our UX department as a business, with responsibilities, requirements, and expectations."

Vasileios Xanthopoulos

A Top-Down and Bottom-Up Approach to User-Centric Maturity at Scale

January 8, 2024

Andrew Custage

"Younger visitors are nearly twice as likely to engage with digital chat or help but are less satisfied afterward."

Andrew Custage Michael Mallett

The Digital Journey: Research on Consumer Frustration and Loyalty

March 29, 2023

Marjorie Stainback

"We had one researcher for every 10 designers, sometimes even one for every 15."

Marjorie Stainback Kelsey Kingman

Transforming Strategic Research Capacity through Democratization

October 24, 2019

Alla Weinberg

"Work gets done through our relationships, and that becomes more true the higher you go in your career."

Alla Weinberg

Cross-Functional Relationship Design

December 6, 2022

Jerome “Axle” Brown

"Caring for people emotionally during insight sessions directly impacts their productivity and willingness to engage."

Jerome “Axle” Brown

How to Use Self-Directed Learning to Ensure Your Research Insights are Heard and Acted Upon

March 11, 2021

Jen Crim

"Our offices have stand-up desks, nice collaboration areas, and comfy seating with a fresh, on-brand look."

Jen Crim Jess Quittner Saritha Kattekola Alex Karr Gurbani Pahwa

Culture, DIBS & Recruiting

June 11, 2021