Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
Have fun with statistics?
Summary
Let’s face it, many of us feel daunted by statistics. But we also know that colleagues and clients ask whether our research has “statistically significant” results. Erin’s book Design for Impact helps you to test your hypotheses about improving design, and she guides you through deciding on your effect sizes to help you get those statistically significant results. Caroline’s book Surveys That Work talks about “significance in practice” and she’s not all that convinced about whether it’s worth aiming for statistical significance. Watch this lively session where Erin and Caroline compared and contrasted their ideas and approaches - helped by your questions and contributions.
Key Insights
-
•
Statistical significance often confuses practitioners because it requires mentally flipping hypotheses and disproving nulls, which is cognitively demanding.
-
•
Effect size is critical to understanding whether a change detected by statistics is meaningful in practice, a concept often neglected in statistics education.
-
•
Fast progress isn't necessarily good progress; teams benefit from slowing down and using statistics to ensure they're moving in the right direction.
-
•
Engineers can be reluctant to implement experiments due to the extra coding load, but they respond well when they understand the learning value gained.
-
•
Survey results (the numerical outcomes) are often confused with the number of respondents required for statistical significance, leading to misunderstandings.
-
•
Statistical thresholds like 95% confidence can be adjusted depending on project needs; lower confidence levels are sometimes acceptable.
-
•
A good hypothesis often starts as an intuitive guess, which gets refined over time through repeated testing and data collection.
-
•
Qualitative and quantitative research should be viewed as complementary tools in a holistic research approach rather than opposed methods.
-
•
AI tools can help generate first drafts of survey questions, but human-centered pilot testing is essential to avoid errors and misinterpretations.
-
•
It's common and acceptable to act on results that are significant in practice but not statistically significant, especially when outcomes clearly affect users.
Notable Quotes
"Statistics is hard because you have to flip flop in your head: think of a hypothesis, then a null hypothesis, then try to disprove the null."
"People confuse statistical significance with significance in practice — they want to know if the change is meaningful, not just mathematically significant."
"Fast is not a virtue in and of itself; moving slower and acting with intention ensures you go in the right direction."
"Engineers hate writing more code, so getting them to buy into experiments means showing the value of the learning on the other side."
"You can have an effect size that matters in practice but isn’t statistically significant, like five users failing a key task in usability testing."
"Most science starts with somebody pulling a number out of their ass — it’s okay to start with a gut instinct or guess."
"Statistics is another tool in our toolbox, part of a hierarchy of evidence that includes qualitative and quantitative methods."
"AI can create first drafts of survey questions, but unless you pilot test with real humans, you won’t know if your audience gets it."
"A lot of people think 95% confidence is the only way, but you can adjust confidence levels based on your situation and needs."
"Start with basics like means, minimums, and ranges — statistics rapidly becomes less mysterious and more useful with practice."
Or choose a question:
More Videos
"Standardization lives deep in my heart partially because of the teams that I’ve worked on and also because I’ve just had so many domains under the umbrellas of my teams over time."
Candace MyersStandardizing Design at Scale
September 9, 2022
"Go find a partner in Ops who has a passion for operations and pain points of inefficiency, then build something great together."
Sarah Sgarlato Pierini Kevin NewtonFrom Passion to Execution: A Story of Evolving Research Maturity at LinkedIn
September 9, 2022
"Technology does not have a mind of its own; people with power set its trajectory and we can influence that."
Cennydd Bowles Dan Rosenberg Lisa WelchmanDay 1 Panel
June 4, 2024
"Sometimes you have to fill a need even if there’s no formal name or structure for it yet."
Rob Mitzel Sébastien MaloThe Tale of Two Companies: Building a Successful UX Practice in a Century-Old Enterprise
January 8, 2024
"If knowledge is difficult to find or hard to use, then all the great information doesn’t really help anyone."
Maria TaylorKnowledge is Power: Managing the Lifeblood of the Design Org
October 3, 2023
"Virtual personas give you directional feedback when human user testing isn't feasible."
Joe Meersman Pooria SohiUse AI to Drive Outcomes that Go Beyond the Design Sprint
September 25, 2024
"We don’t use yesterday’s retail playbook to solve problems for today — that’s impossible."
Sara Asche Anderson Jamie KaspszakNot Your Ordinary Re-Brand: Design's Path to Driving Customer Obsession at Best Buy
January 8, 2024
"When I see users saying, oh my goodness, I didn’t know it did that, that’s a clear sign of too much complexity."
David Cronin Uday Gajendar Peter Morville Kendra ShimmellDiscussion
May 13, 2015
"Our call to action is to co-create these measurement standards together and share back to the community."
Kristin Skinner Kamdyn MooreGroup Activity: A Deep Dive Into Value and Outcomes
October 23, 2019
