Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Quantitative instruments are frequently sought because 1) they can be quickly fielded to lots and lots of people, and 2) when carefully sampled, they can be generalizable to the population of users/customers. However, because many times the focus is on speed to launch because decision-makers need results quickly, there is not much depth given to their development, nor an investigation of the validity evidence. In the session, I will share a framework that centers validity and is necessarily a mixed methods approach to research. I will also share ideas on how to scale the research over time so that findings and insights are able to be iteratively delivered to stakeholders, while also iteratively informing one another in a qual-quant research dance that brings more trustworthy, user-centered evidence to decision-makers. Finally, I will share ideas for a course I am developing for supporting qualitative researchers to become more mixed in their approach.
Key Insights
-
•
A mixed-methods approach enhances validity and trust in survey research.
-
•
Validity is crucial in both qualitative and quantitative research to ensure accurate interpretations.
-
•
Engaging stakeholders early in the survey design process leads to better alignment with their needs.
-
•
Cognitive interviews can uncover participant understanding and response processes, refining survey items.
-
•
Iterative testing of surveys allows for continuous improvement and insight generation over time.
-
•
It's important to address both the ethical implications of survey questions and their potential impact on participants.
-
•
Researcher collaboration between quant and qual experts strengthens the validity argument.
-
•
Surveys should be treated as products requiring careful design and testing before deployment.
-
•
Identifying and understanding subpopulations can reveal biases and improve survey accuracy.
-
•
Validity assessments should be continuous, reflecting changes in user bases and research goals.
Notable Quotes
"I believe that when we have these multiple points of information, it does help build trust and improves our decision-making processes."
"Validity is the degree to which evidence and theory support the interpretations of test scores for proposed uses."
"When you have broader research goals that overlap, they can support you in answering important questions."
"These surveys are typically used to answer questions that support multiple stakeholder interests."
"If you're only doing qualitative research, you should still consider how you can contribute to the validity argument."
"Observing how people interpret questions can provide significant insights into survey design."
"Test content requires spending time talking with stakeholders about desired outcomes."
"Reliability and validity must work together to create meaningful surveys."
"Ethics in survey research should be top of mind throughout all stages of the process."
"We are the users of this product, and we need to iteratively test it."
More Videos
"Don't try to solve for the system, try to understand the request."
Jacqui FreyFlow and Superfluidity for Design Orgs
November 7, 2018
"It's not just the mind that needs communication to thrive, it's also our very human hearts."
Laura Gatewood Laine ProkayBeyond Buzzwords: Adding Heart to Effective Slack Communication
September 23, 2024
"The product leadership doesn't see the value in research and they're thinking about shutting the research team down."
Johanna KollmannInsights-Driven Product Strategy: Get your Research to Count
December 6, 2022
"Good morning, good evening and afternoon everybody."
Bria AlexanderDay 3 Welcome
September 25, 2024
"Throw away the rule book—just try to get as many ideas as you can out as fast as possible."
Billy CarlsonIdeation tips for Product Managers
December 6, 2022
"It's easy to stay immersed in digital, but physical contexts matter immensely for designing multi-channel experiences."
Catherine DubutBridging Physical and Digital Spaces: Approaches to Retail Service Design
March 18, 2021
"Onboarding processes should empower new hires to integrate smoothly into the team."
Kim Holt Emma Wylds Pearl Koppenhaver Maisee XiongA Salesforce Panel Discussion on Values-Driven DesignOps
September 8, 2022
"What's the last bad experience you had with another discipline?"
Adrian HowardSturgeon’s Biases
September 25, 2024
"I taught myself graphic design and HTML and CSS, and that choice to be on Neopets in my space at the time really launched my career as a designer."
Asia HoePartnering with Product: A Journey from Junior to Senior Design
November 29, 2023
