Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.
Key Insights
-
•
A mixed-methods approach enhances validity and trust in survey research.
-
•
Validity is crucial in both qualitative and quantitative research to ensure accurate interpretations.
-
•
Engaging stakeholders early in the survey design process leads to better alignment with their needs.
-
•
Cognitive interviews can uncover participant understanding and response processes, refining survey items.
-
•
Iterative testing of surveys allows for continuous improvement and insight generation over time.
-
•
It's important to address both the ethical implications of survey questions and their potential impact on participants.
-
•
Researcher collaboration between quant and qual experts strengthens the validity argument.
-
•
Surveys should be treated as products requiring careful design and testing before deployment.
-
•
Identifying and understanding subpopulations can reveal biases and improve survey accuracy.
-
•
Validity assessments should be continuous, reflecting changes in user bases and research goals.
Notable Quotes
"I believe that when we have these multiple points of information, it does help build trust and improves our decision-making processes."
"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."
"When you have broader research goals that overlap, they can support you in answering important questions."
"These surveys are typically used to answer questions that support multiple stakeholder interests."
"If you're only doing qualitative research, you should still consider how you can contribute to the validity argument."
"Observing how people interpret questions can provide significant insights into survey design."
"Test content requires spending time talking with stakeholders about desired outcomes."
"Reliability and validity must work together to create meaningful surveys."
"Ethics in survey research should be top of mind throughout all stages of the process."
"We are the users of this product, and we need to iteratively test it."
More Videos
"Complexity is not a barrier for accessibility; modern assistive tech is highly advanced."
Samuel ProulxInvisible barriers: Why accessible service design can’t be an afterthought
December 3, 2024
"Live, listen, laugh, and learn while being open to transformation."
Victor UdoewaTheme One Intro
March 27, 2023
"It doesn't matter how big the company is; we can always start from a very simple context."
Gonzalo GoyanesDesign ROI: Cover a Little, Get a Lot
September 8, 2022
"It's hard to predict what harms will emerge from new technologies."
Alexandra SchmidtWhy Ethics Can't Save Tech
November 18, 2022
"We have to be relentless at saying no to avoid overcommitment."
Jose CoronadoFrom Zero to Hero
September 8, 2022
"All neurodiversity is unique and not one size fits all."
Pippa LomasPaving the Path for Neurodiversity in Design
October 4, 2023
"We shouldn’t be afraid to smuggle design thinking into conversations about climate policy."
Louis Rosenfeld Matt Jones Olga Khroustaleva Michael Leggett Karol MunozDo you want to work on climate? (Climate UX Discussion Series)
November 15, 2023
"If you don't hire designers, you're still gonna get design. It's just gonna suck."
Jeff GothelfThe Intersection of Lean and Design
January 10, 2019
"In organizations, the people are at the center of everything we're trying to evolve."
Dave GrayGroup Activity: Making Sense of DesignOps
November 7, 2017
