Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Summary
Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.
Key Insights
-
•
Statistics can be intimidating due to its mathematical language and cultural perceptions in the design community.
-
•
Understanding the difference between statistical significance and practical significance is crucial in making informed decisions.
-
•
Effect sizes help determine the meaningfulness of any changes in research outcomes, highlighting the importance of measuring bigger impacts rather than just statistical significance.
-
•
Moving fast is not always synonymous with progress in product design; thoughtful research leads to better decisions.
-
•
Developing an intuitive understanding of user behavior takes time and experimentation, which helps make better guesses about outcomes.
-
•
The use of statistics in research should not be seen as a barrier but rather as a vital tool to enhance decision-making processes.
-
•
Bad survey questions can yield misleading results, illustrating the need for precision in formulation.
-
•
Collaborating with engineers requires integrating their logic into your statistical methods to promote understanding and adoption of research practices.
-
•
Research should be holistic – integrating qualitative and quantitative data enhances overall insights.
-
•
Practicing basic statistical concepts, like averages, helps build comfort and familiarity with numbers.
Notable Quotes
"Statistics can feel hard because we're using math to understand the world around us."
"We want to make progress fast, but we must ensure we're heading in the right direction."
"Fast is not a virtue in and of itself; it's about acting with intention."
"You can have something that's significant in practice but not statistically significant – and vice versa."
"The real question is whether a change is meaningful and helpful, not just if it’s statistically significant."
"Statistical significance is a number that tells us whether the change happened by chance or due to our actions."
"Not all numbers need to be precise; they just need to provide a useful understanding of the situation."
"If you want to ship faster, know what you're aiming for and if that aim is worth pursuing."
"You get better results from asking one person a useful question than asking 10,000 people a silly one."
"The magic of statistics comes from how it can help predict what's true and what's real."
More Videos
"We help organizations build relationships with their audiences just like people do."
Noz UrbinaRapid AI-powered UX (RAUX): A framework for empowering human designers
May 1, 2025
"We can improvise the rest on top of that."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
June 4, 2019
"I am not someone who is in any way a proponent of a design challenge."
Holly ColeUnderstanding Experiences: When you have to do more than work
November 8, 2018
"Hello, everyone, my name is Jack Behar and I will be leading this session."
Jack BeharHow to Build Prototypes that Behave like an End-Product
December 6, 2022
"You don't want to miss out on one of the really cool parts about the conference, which is going home with a bunch of cool swag."
Bria AlexanderOpening Remarks
October 3, 2023
"We are the ground control, building our systems gradually to reach our organizational 'moon'."
Erika FlowersAI-Readiness: Preparing NASA for a Data-Driven, Agile Future
June 10, 2025
"Teachers need actionable insights, not just raw data."
Kristin SkinnerFive Years of DesignOps
September 29, 2021
"Big ears mean listening to other people more than you're listening to yourself."
Jim KalbachJazz Improvisation as a Model for Team Collaboration
November 6, 2017
"There's so much creative talent in 8 billion creative humans on planet Earth."
Surya VankaUnleashing Swarm Creativity to Solve Enterprise Challenges
June 10, 2021
