Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

A Mixed Method Approach to Validity to Help Build Trust
Friday, April 28, 2023 • QuantQual Interest Group
Share the love for this talk
A Mixed Method Approach to Validity to Help Build Trust
Speakers: Chris Engledowl
Link:

Summary

Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.

Key Insights

  • Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.

  • Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.

  • Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.

  • Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.

  • Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.

  • Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.

  • Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.

  • Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.

  • Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.

  • It is important to revisit validity considerations continuously, especially after product or user base changes.

Notable Quotes

"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."

"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."

"What would happen if you do think about validity? Would it change your process or research plans?"

"Surveys are products too, so they need to be iteratively tested."

"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."

"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."

"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."

"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."

"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."

"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."

Ask the Rosenbot
Matt Webb
Context Window: Five Futures for AI
2025 • Designing with AI 2025
Gold
Ellie Krysl
Planned Right. Managed Right. Designed Right.
2023 • Enterprise UX 2023
Gold
Doug Powell
Closing Keynote: Design at Scale
2018 • DesignOps Summit 2018
Gold
Sam Proulx
Online Shopping: Designing an Accessible Experience
2023 • Enterprise UX 2023
Gold
Brian T. O’Neill
Does Designing and Researching Data Products Powered by ML/AI and Analytics Call for New UX Methods?
2022 • QuantQual Interest Group
Sam Proulx
To Boldly Go: The New Frontiers of Accessibility
2022 • Design at Scale 2022
Gold
Tricia Wang
The most popular design thinking strategy is BS
2022 • Enterprise Community
Bria Alexander
Opening Remarks
2022 • Civic Design 2022
Gold
Lori Muszynski
Keeping Design Weird
2023 • DesignOps Summit 2023
Gold
Dantley Davis
Leadership & Diversity—A Fireside Chat with Dantley Davis
2020 • Enterprise Community
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Yunyan Li
UX Best Practices
2021 • Design at Scale 2021
Gold
Saara Kamppari-Miller
Inclusive Design is DesignOps
2021 • DesignOps Summit 2021
Gold
Shreya Dhawan
Making service tangible: the fastest path to higher performance
2025 • Advancing Service Design 2025
Gold
Sarah Gallimore
Inspire Progress with Artifacts from the Future
2022 • Civic Design 2022
Gold
Pippa Lomas
Paving the Path for Neurodiversity in Design
2023 • DesignOps Summit 2023
Gold

More Videos

George Aye

"Power is the ability to change another person’s reality."

George Aye

That Quiet Little Voice: When Design and Ethics Collide

November 16, 2022

Xenia Adjoubei

"We fundraised to cover Tatiana’s salary for three months so she could launch this project very quickly after the war started."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Benjamin Wiedmaier

"The only way to do research wrong is to not do it at all."

Benjamin Wiedmaier Annie Mayfield

Redefining Toolkits: Unbundling to Create a Perfect Match

March 11, 2025

John Devanney

"Not every head of design is the right leader for every phase of organizational maturity."

John Devanney

The Design Management Office

November 6, 2017

Steve Sanderson

"We hired designers who speak product, think in hypotheses, and love constraints rather than just artistic ideas."

Steve Sanderson Alissa Briggs Jeff Gothelf Bill Scott

Discussion

May 14, 2015

Jemma Ahmed

"Jackie Ho talks about how to keep teams accountable and aligned while maintaining their autonomy."

Jemma Ahmed

Theme 2 Intro

January 8, 2024

Megan Kierstead

"Confidence doesn’t dull criticality; self-doubt is a statement about your own worth, which we want to reduce."

Megan Kierstead

You Are a Badass at UX: Overcoming Imposter Syndrome

March 10, 2021

PJ Buddhari

"Instead of choosing colors then checking contrast, we flipped the process and started by defining contrast ratio targets and then generating colors to meet them."

PJ Buddhari Nate Baldwin

Meet Spectrum, Adobe’s Design System

June 9, 2021

James Lang

"We often introduce friction deliberately in community building to make it safe and meaningful."

James Lang

If you can design an app, you can design a community

May 22, 2025