Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Integrating Qualitative and Quantitative Research from Discovery to Live
Friday, December 16, 2022 • QuantQual Interest Group
Share the love for this talk
Integrating Qualitative and Quantitative Research from Discovery to Live
Speakers: Mackenzie Cockram , Sara Branco Cunha and Ian Franklin
Link:

Summary

Over the last three years Southampton University in the UK has been doing a complete website redesign following an Agile process; user research and performance analytics have been an integral part of the Agile process from the very start. The website has a variety of products aimed at potential students and research collaborators. Performance analytics informed the business case and objectives and qualitative research in Discovery uncovered the user needs to improve the user experience. During alpha qualitative research informed the design of the early prototypes and in beta analytics and user research integrated qual and quant in a variety of metrics around performance and the user experience. The same metrics have been taken forward and enhanced in Live to ensure continuous improvement which sits alongside the new product roadmap. The presentation will outline the integration of qual and quant and give examples of what has been done, the metrics, and how they are informing the user experience and business objectives.

Key Insights

  • The University of Southampton faced over 4 million URLs due to decentralized content, causing poor user experience and high bounce rates.

  • Replacing 1000+ web content authors with a skilled specialist team improved content quality, consistency, and sustainability.

  • Integrating quantitative analytics with qualitative user research enabled a comprehensive understanding of user journeys and needs.

  • Measuring over 30 distinct user journeys allowed targeted benchmarking and direct evaluation of site performance.

  • Adopting user-centered design was a cultural shift from the university’s previous waterfall approach.

  • The beta phase attracted over 4,000 users, providing rich data to validate design choices before full launch.

  • Heat maps and recordings revealed user behaviors like 'click rage' where users clicked on non-clickable elements.

  • Microsoft reaction cards helped capture user sentiment efficiently, complementing free-text feedback.

  • Continuous improvement is enabled by automated data analysis using R scripts and a process to triage issues for design sprints or backlog.

  • Stakeholder involvement in observing real user sessions is crucial to overcoming resistance and building empathy for user-centered design.

Notable Quotes

"We are not collecting vanity analytics, the data is doing real work on informing design."

"85% of people were leaving after viewing only one page, showing severe bounce issues."

"The new site had a 19% dropout rate compared to 32% on the old site, showing clear improvement."

"User journeys are the bedrock for deciding which data to collect and measure."

"Some stakeholders still prefer their beliefs to data, even in academia."

"Heat maps showed users clicking on areas they thought were clickable but weren’t, resulting in click rage."

"We brought all the user journeys together with UX designers, content designers, product owners — it wasn’t just analytics and research in isolation."

"Having the qualitative and quantitative data together lets us tell a comprehensive story to stakeholders."

"We started analytics at the beginning so we could benchmark and compare old and new site performance."

"Agile can feel like a cult, but the data helps show it’s working and not just a belief system."

Ask the Rosenbot
John Taschek
Making People the X-Factor in the Enterprise
2018 • Enterprise Experience 2018
Gold
Mila Kuznetsova
How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices
2022 • Advancing Research 2022
Gold
Amy Thibodeau
Opening Keynote: Process and Ambiguity
2019 • DesignOps Summit 2019
Gold
Sabrina Mach
How to Design Your Design Operating Model
2021 • DesignOps Summit 2021
Gold
Bria Alexander
Opening Remarks Day 1
2024 • Advancing Research 2024
Gold
Angy Peterson
More Than Technology: Personalized Public Sector Experiences
2021 • Civic Design 2021
Gold
Margot Bloomstein
Fostering Trust in Your Brand and Beyond
2020 • Enterprise Community
Louis Rosenfeld
Coffee with Lou #3: What Makes for a Successful UX Conference Presentation?
2024 • Rosenfeld Community
Crystal Philcox
The Many Faces of Operations
2017 • DesignOps Summit 2017
Gold
Boon Yew Chew
Making Sense of Systems—and Using Systems to Make Sense of the Enterprise
2023 • Enterprise UX 2023
Gold
Alana Washington
Theme 1: Introduction and Provocation
2024 • DesignOps Summit 2020
Gold
Katie Hansen
Finding the unknown in the known: Harnessing meta-analysis and literature review
2025 • Advancing Research 2025
Gold
Laine Riley Prokay
How DesignOps can Drive Inclusive Career Ladders for All
2021 • DesignOps Summit 2021
Gold
Deanna Washington
Connecting the Ops: Plenary Panel and Closing Circle
2022 • DesignOps Summit 2022
Gold
Ryan Matthew
Bridging Design and Code: AI-Powered Design System Integration
2025 • Rosenfeld Community
Adel Du Toit
Get Your CFO To Say: 'Our Strategic Goal is User Obsession'
2022 • Design at Scale 2022
Gold

More Videos

Carol Scott

"Trauma is a universal human experience."

Carol Scott Melissa Eggleston

Avoid Harming Your Team and Users: Promoting Care and Brand Reputation with Trauma-Informed UX Practices

February 5, 2025

Lija Hogan

"The era of 100-page reports is ending because we simply can’t produce them fast enough at scale."

Lija Hogan Milan Mijatovic Sam Proulx Louis Rosenfeld

Three Years Out: Perspectives on the Near-Term Future of User Research

March 15, 2024

Sam Proulx

"If you want an accessible app on iOS and Android, using the standard native APIs and controls will get you there."

Sam Proulx

Mobile Accessibility and You

June 9, 2022

Ana Ferreira

"We evaluate the work and not the amount of hours."

Ana Ferreira

Designing Distributed: Leading Doist’s Fully Remote Design Team in Six Countries

January 8, 2024

Jorge Arango

"Working with these tools requires new workflows and checkpoints to validate and tweak the AI's output."

Jorge Arango

Scale Smart: AI-Powered Content Organization Strategies

September 24, 2024

Chris Moses

"The first year federal incentives ended, the market changed completely; buyers started listening to end users more."

Chris Moses

Stretching the Definition of DesignOps with Product Development

November 7, 2018

Trisha Causley

"A prompt is just a set of instructions to an AI in the context of text generation."

Trisha Causley

[Demo] Complexity in disguise: Crafting experiences for generative AI features

June 5, 2024

Clara Kliman-Silver

"It’s critical to measure perceived value and ask: Are you really solving the right problem with AI?"

Clara Kliman-Silver

UX Futures: The Role of Artificial Intelligence in Design

June 7, 2023

Michael Polivka

"Design is frequently misunderstood; sometimes it’s just about explaining why it’s important to get others involved."

Michael Polivka

Scaling Design through Relationship Maps

November 7, 2017