Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
A Mixed Method Approach to Validity to Help Build Trust
Summary
Key Insights
-
•
Validity involves five evidence sources: content, response processes, internal structure, relations to other variables, and consequences of testing.
-
•
Qualitative methods, especially cognitive interviews, are crucial for understanding how respondents interpret survey items, supporting validity.
-
•
Surveys should be treated as products that need ongoing iteration and testing, not one-off tools.
-
•
Ethical considerations extend beyond data privacy to how survey results affect user experience and product decisions.
-
•
Mixed methods approaches leverage both qualitative insights and quantitative analyses to build a stronger validity argument.
-
•
Breaking down survey validation work across multiple teams and 'bite-sized' efforts makes the process manageable.
-
•
Revising surveys over time to improve validity complicates measuring change longitudinally but increases trustworthiness.
-
•
Stakeholder buy-in improves when validity processes are communicated as phased insights offering tangible results quickly.
-
•
Analyses such as factor analysis and Rasch modeling reveal survey internal structure and help identify item bias across subpopulations.
-
•
It is important to revisit validity considerations continuously, especially after product or user base changes.
Notable Quotes
"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."
"Qualitative research in a mixed methods setting needs to think about validity to support the bigger validity argument."
"What would happen if you do think about validity? Would it change your process or research plans?"
"Surveys are products too, so they need to be iteratively tested."
"You don’t know what you don’t know. Let’s write surveys that cover those blind spots."
"If the survey wording changes, measuring change over time becomes difficult, but improving the survey builds trust."
"Ethical considerations should go beyond typical privacy reviews and think deeply about user impact throughout the process."
"It is better to partner with someone who has quantitative expertise to interpret internal structure analyses."
"Management and stakeholders often care more about getting usable information quickly than understanding the full validity process."
"We can get quick insights in phases to keep teams engaged and slowly build a complete validity argument."
Or choose a question:
More Videos
"Being humble enough to ask a competitor for feedback changed everything and led directly to job offers."
Corey LongHiring in DesignOps: A Critical Study on How to Hire and Get Hired
September 23, 2024
"Pay transparency within organizations is critical for enabling good work and healthier working conditions."
Sarah Auslander Betsy Ramaccia Gordon RossInsights Panel
November 18, 2022
"Human biases are the real problem behind algorithmic bias because organizations embed those biases first."
Lisa WelchmanCleaning Up Our Mess: Digital Governance for Designers
June 14, 2018
"We must create spaces that allow nature to thrive within our cities."
Lena ShenkarenkoCollaborative Wireframing for Creating Team Alignment and Shipping Better Products
October 21, 2020
"We have to design not just for humans but for other life forms on this planet."
Christian BasonExpand—Rethinking Design for Public Challenges
September 14, 2022
"Social opportunities to connect with each other actually offset the remote fatigue and isolation that many were feeling."
Nicole UmphressDelivering Design Education During a Global Pandemic: Lessons Learned
June 9, 2022
"Empathy in jazz means the band is in it together—when someone plays a wrong chord, the rest adapt and turn it into an opportunity."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
November 6, 2017
"Building relationships with our internal partners may well be the highest leverage aspect of our jobs."
Bob BaxleyTheme 4: Intro
January 8, 2024
"If seeing a diverse participant pool is important to you, it's your responsibility to make sure your recruiters use it as a priority."
Megan CamposWhat Did I Miss? The Hidden Costs of Deprioritizing Diversity in User Research
March 12, 2021
Latest Books All books
Dig deeper with the Rosenbot
What role does documenting accessibility trade-offs play in future-proofing design systems for AI?
How can breaking down intractable problems into smaller visual chunks make them manageable?
What advice do Julia and Clay give for transitioning into product management from design or other adjacent roles?