Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Integrating Qualitative and Quantitative Research from Discovery to Live
Friday, December 16, 2022 • QuantQual Interest Group
Share the love for this talk
Integrating Qualitative and Quantitative Research from Discovery to Live
Speakers: Mackenzie Cockram , Sara Branco Cunha and Ian Franklin
Link:

Summary

Over the last three years Southampton University in the UK has been doing a complete website redesign following an Agile process; user research and performance analytics have been an integral part of the Agile process from the very start. The website has a variety of products aimed at potential students and research collaborators. Performance analytics informed the business case and objectives and qualitative research in Discovery uncovered the user needs to improve the user experience. During alpha qualitative research informed the design of the early prototypes and in beta analytics and user research integrated qual and quant in a variety of metrics around performance and the user experience. The same metrics have been taken forward and enhanced in Live to ensure continuous improvement which sits alongside the new product roadmap. The presentation will outline the integration of qual and quant and give examples of what has been done, the metrics, and how they are informing the user experience and business objectives.

Key Insights

  • The University of Southampton faced over 4 million URLs due to decentralized content, causing poor user experience and high bounce rates.

  • Replacing 1000+ web content authors with a skilled specialist team improved content quality, consistency, and sustainability.

  • Integrating quantitative analytics with qualitative user research enabled a comprehensive understanding of user journeys and needs.

  • Measuring over 30 distinct user journeys allowed targeted benchmarking and direct evaluation of site performance.

  • Adopting user-centered design was a cultural shift from the university’s previous waterfall approach.

  • The beta phase attracted over 4,000 users, providing rich data to validate design choices before full launch.

  • Heat maps and recordings revealed user behaviors like 'click rage' where users clicked on non-clickable elements.

  • Microsoft reaction cards helped capture user sentiment efficiently, complementing free-text feedback.

  • Continuous improvement is enabled by automated data analysis using R scripts and a process to triage issues for design sprints or backlog.

  • Stakeholder involvement in observing real user sessions is crucial to overcoming resistance and building empathy for user-centered design.

Notable Quotes

"We are not collecting vanity analytics, the data is doing real work on informing design."

"85% of people were leaving after viewing only one page, showing severe bounce issues."

"The new site had a 19% dropout rate compared to 32% on the old site, showing clear improvement."

"User journeys are the bedrock for deciding which data to collect and measure."

"Some stakeholders still prefer their beliefs to data, even in academia."

"Heat maps showed users clicking on areas they thought were clickable but weren’t, resulting in click rage."

"We brought all the user journeys together with UX designers, content designers, product owners — it wasn’t just analytics and research in isolation."

"Having the qualitative and quantitative data together lets us tell a comprehensive story to stakeholders."

"We started analytics at the beginning so we could benchmark and compare old and new site performance."

"Agile can feel like a cult, but the data helps show it’s working and not just a belief system."

Ask the Rosenbot
Farid Sabitov
Automatization for Large Enterprise Teams
2024 • DesignOps Summit 2020
Gold
Louis Rosenfeld
The Bigger Picture: A Panel Discussion
2019 • DesignOps Summit 2019
Gold
Daniela Magaña Flores
Ahead of Competition: Learn What UX Benchmarking Can Do for Your Business Today
2022 • Advancing Research 2022
Gold
Kristen Honey
"Let’s Talk About Data and Crisis”: Public Digital Service Delivery = Open Data + Human Centered Design
2021 • Civic Design Community
Angelos Arnis
Navigating the Rapid Shifts in Tech's Turbulent Terrain
2023 • DesignOps Summit 2023
Gold
Mariesa Lenz
What Beekeeping Taught me about Product Teams
2025 • Rosenfeld Community
Robin Beers
Navigating organizational systems: Rethinking researcher’s role in driving change
2025 • Advancing Research 2025
Gold
Stephen Anderson
Puzzled? How to Coordinate Humans for Complex Challenges
2021 • Enterprise Community
Cassandra Piester
Developing and Deploying Your Design Operations Strategy
2024 • DesignOps Summit 2024
Gold
Erin Hauber
Design is Not the Frosting on the Scaled Agile Layer Cake
2019 • DesignOps Summit 2019
Gold
Matt LeMay
You Don’t “Get” Anyone to Do Anything
2022 • Design in Product 2022
Gold
Kritika Yadav
Optimizing AI Conversations: A Case Study on Personalized Shopping Assistance Frameworks
2025 • Designing with AI 2025
Gold
Alla Weinberg
How to Build and Scale Team Safety
2024 • DesignOps Summit 2020
Gold
Dr. Jamika D. Burge
The Future of Research: Bridging the Gaps
2021 • Advancing Research Community
Ren Pope
Building Experiences for Knowledge Systems
2023 • Enterprise UX 2023
Gold
Bria Alexander
State of DesignOps Panel
2021 • DesignOps Summit 2021
Gold

More Videos

Bria Alexander

"Adjust your conference time zone setting depending on where you are to avoid confusion."

Bria Alexander

Opening Remarks

November 18, 2022

Hugh Dubberly

"Digital twins are replicas of dynamic systems that help both individuals and populations manage and improve operations."

Hugh Dubberly

Problems with Problems: Reconsidering the Frame of Designing as Problem-Solving

June 19, 2019

April Reagan

"Design ops professionals are already futurists in disguise because they constantly manage risks and plan capacity."

April Reagan

Look, Think, Act: The Futures-Smart Design Organization

October 1, 2021

Ricardo Martins

"Conjoint analysis was used by Apple and Samsung to determine the financial value of specific product features."

Ricardo Martins

Unlocking the power of advanced quantitative methods

March 12, 2025

Maria Skaaden

"How can developers and designers work together to create great products? That’s the question."

Maria Skaaden

Continuous Design: One eye on the horizon and the other on the next wave

November 8, 2018

Vanessa Varin

"Match the moment. Calibrate the altitude and the type of feedback to the context."

Vanessa Varin

Feedback: The Other F-Word

September 10, 2025

Jen Crim

"We have maintained that freshness and acceptance of new ideas while growing rapidly."

Jen Crim Jess Quittner Saritha Kattekola Alex Karr Gurbani Pahwa

Culture, DIBS & Recruiting

June 11, 2021

Patrick Boehler

"Finding meaningful insights is not just casting a wide net—it requires discipline, structure, and knowing where to fish."

Patrick Boehler

Fishing for Real Needs: Reimagining Journalism Needs with AI

June 10, 2025

Julie Baher

"Peter Conner, our lawyer, redesigned compliance training so people took fewer courses but more relevant ones."

Julie Baher

Culture Change—My Journey

May 14, 2015