Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust
Friday, April 28, 2023 • QuantQual Interest Group
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.

  • Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.

  • Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.

  • Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.

  • Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.

  • Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.

  • Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.

  • Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.

  • Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.

  • It is important to revisit validity considerations continuously, especially after product or user base changes.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."

"What would happen if you do think about validity? Would it change your process or research plans?"

"Surveys are products too, so they need to be iteratively tested."

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."

"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."

"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."

"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."

"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."

Ask the Rosenbot
Amy Bucher
Harnessing behavioral science to uncover deeper truths
2025 • Advancing Research 2025
Gold
Raven Veal
Dark Metrics: Illuminating the Negative Impact of Digital Health Design
2021 • Advancing Research 2021
Gold
Noreen Whysel
Short Take #4: UX/Product Lessons from Your Industry Peers
2022 • Design in Product 2022
Gold
Lada Gorlenko
Theme 1: Intro
2024 • Enterprise Experience 2020
Gold
Aletheia Delivre
New Shapes and Emerging Identities for Design Ops
2025 • DesignOps Summit 2025
Gold
Christian Rohrer
Research Operations at Scale
2017 • DesignOps Summit 2017
Gold
Spencer L. A. Stultz
Why Social Justice Frameworks are Necessary for Successful DEI/JEDI Initiatives
2023 • DesignOps Summit 2023
Gold
Bas Raijmakers, PhD (RCA)
What Design Research can Learn from Documentary Filmmaking
2022 • Advancing Research 2022
Gold
Etienne Fang
The Power of Care: From Human-Centered Research to Humanity-Centered Leadership
2021 • Advancing Research 2021
Gold
Joseph Williams
Unlocking impact and influence through inclusive hiring in research
2021 • Advancing Research Community
Mackenzie Guinon
M.C. Escher’s UX Research Career Ladder
2022 • Advancing Research 2022
Gold
Mohammad Hossein Jarrahi
Contextuality problem: Exploring the Benefits of Qualitative and Quantitative Research
2023 • QuantQual Interest Group
Sarah Rink
Remote User Research: Dos and Don'ts from the Virtual Field
2020 • Advancing Research Community
Jon Fukuda
All the Ops: Successful cross-functional collaboration
2025 • DesignOps Summit 2025
Gold
Vincent Brathwaite
Opener: Past, Present, and Future—Closing the Racial Divide in Design Teams
2020 • DesignOps Summit 2020
Gold
Anat Fintzi
Delivering at Scale: Making Traction with Resistant Partners
2022 • Design at Scale 2022
Gold

More Videos

Doug Powell

"Good design is good business."

Doug Powell

Closing Keynote: Design at Scale

November 8, 2018

Mila Kuznetsova

"Make sure children understand they can stop the session at any time, and confirm they really get that."

Mila Kuznetsova Lucy Denton

How Lessons Learned from Our Youngest Users Can Help Us Evolve our Practices

March 9, 2022

Sarah Gallimore

"Artifacts of the future help us envision and grasp the change that we’re hoping to have—it’s not just bullet points or vision statements."

Sarah Gallimore

Inspire Progress with Artifacts from the Future

November 18, 2022

Lada Gorlenko

"We never compromise on the business goals, but how we achieve them can be flexible and pragmatic."

Lada Gorlenko Sharbani Dhar Sébastien Malo Rob Mitzel Ivana Ng Michal Anne Rogondino

Theme 1: Discussion

January 8, 2024

Alnie Figueroa

"Start tinkering, playing around, and pushing AI’s limits because the industry shift is going to get faster."

Alnie Figueroa

The Future of Design Operations: Transforming Our Craft

September 10, 2025

Landon Barnes

"Mixed methods research, combining qualitative and quantitative, gives the fullest understanding of customer experience."

Landon Barnes

Are My Research Findings Actually Meaningful?

March 10, 2022

Emily Eagle

"It’s the worst thing when someone feels dumb in front of a customer because the tools don’t support them."

Emily Eagle

Can't Rewind: Radio and Retail

June 3, 2019

Malini Rao

"We neglected our power users during the responsive, one-size-fits-all strategy, and it really hurt us."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Asia Hoe

"Project management was focused on outputs, assuming the correct feature was already defined."

Asia Hoe

Partnering with Product: A Journey from Junior to Senior Design

November 29, 2023