UNcomplicated Analytics | Fantasy Football has a Sample Size Problem

In the scientific community, small sample sizes are a constant problem investigators face. For example, in some neurological experiments, sample sizes of 22 and 24 were found to need to be at least 6 times larger to produce scientifically significant results, with several of these experiments having a recommended sample size in the 400-plus range to be representative of the population (Button et. al, 2013).

Welcome to UNcomplicated Analytics

No, you didn’t accidentally click on a scientific journal article; you’re still on the Undroppables. The scientific community just has a firmer grasp of statistics and biases that negatively affect data than the fantasy football community does. While the populations that the aforementioned studies refer to are a lot larger than what we use in fantasy football, the principles still apply to the football data we are looking at. Advanced statistics and analytics have massively increased in popularity in the NFL, especially in the fantasy football community. Knowing how to interpret these stats, which ones to trust, and which ones to be skeptical of can give the average fantasy football player an advantage that might just help you win your league. My hope is by the end of this article, (and throughout the rest of my series UNcomplicated Analytics) you begin understanding what the statistics you read are really telling you and uncover what they’re not.

Sample Sizes

To dive into why sample size is important, let’s first look at an example. During Week 13 of the 2017 season, the Tennessee Titans beat the Houston Texans 24-13. DeMarco Murray and Derrick Henry split carries for the Titans with 11 carries each. When you look at the box score, you wouldn’t be crazy to wonder why the Titans split carries so evenly. Derrick Henry averaged 9.9 yards per carry to Murray’s 6.0. Based on this data alone, Henry is obviously the more efficient back. He was averaging almost a first down for every rush attempt he had!

Derrick Henry taking it to the house for a 75 yard touchdown #BuiltByBama pic.twitter.com/LOgRfGchgh

— Alabama DieHards (@AlabamaDieHards) December 3, 2017

So why do I bring up a random game from 2017? With 46 seconds left in the game, Derrick Henry had a rush for 75 yards that went for a touchdown. Before that rush, Henry had 34 rush yards on 10 carries, an average of only 3.4 yards per carry. That single carry, the last offensive play by Tennessee in the game, increased his yards per carry by 6.5 yards. Many NFL analysts hate yards per carry for this exact reason, because it’s heavily influenced by outliers that exist within small sample sizes.

Key Takeaways

So now that we have a brief understanding of one of the issues of small sample sizes, how does knowing they’re an issue make us a better fantasy player? Here’s just a few ideas:

RULE ONE

Be very skeptical of small sample sizes, because they can cause us to draw the wrong conclusion. One of the worst offenders of this are splits that you see to highlight a specific player. While these can be great at isolating specific variables and situations, these usually do not tell the whole story. For a quick example, look at the table below:

Wow, the receiver did really poorly with QB2 instead of QB1 so QB2 really was obviously his problem then. So off of one variable, we can conclude that this receiver is bad with QB2. But wait, there is a lot more to this story. If you haven’t figured it out, this is Mike Gesicki’s split with Ryan Fitzpatrick as QB1 and Josh Rosen as QB2. Was Rosen the reason Gesicki played poorly? Maybe. Was it the presence of Preston Williams? That’s also a possibility.

There are a number of variables that can describe why Mike Gesicki was very bad with his two games with Rosen at quarterback, and while Rosen might be the reason, we can’t conclude anything based on such a small sample size. If it actually was the presence of Preston Williams in the offense, (Williams was injured in Week 8 and missed the rest of the season; Rosen started Weeks 2 and 3) you might be higher on Gesicki because you liked his split with Fitzpatrick, only for quarterback play being the wrong variable influencing his play. There is not a quick fix to dealing with small sample sizes, but being aware and skeptical of this type of data will help you from falling for false narratives.

RULE TWO

Be wary of averages, such as yards per carry, yards per reception, and points per game, as these can be heavily influenced by outliers. Our Derrick Henry example from earlier perfectly displays this, but just to drive the idea home, let’s look at Will Fuller’s 2019 season. Changes to the Houston offense aside, if you pull up his profile in a draft this season, you might see an average of 12.2 fantasy points per game in PPR and immediately think he was a solid WR2/3 last season and draft him as such. If this was your way of thinking, you would have completely missed that he scored almost half of his points in a single game, and outside of that game he averaged 8.0 points per game.

Throwback to Week 5 when Will Fuller V put up the NUMBER ONE fantasy performance of the season and the best since Jamaal Charles in 2013.

14 Catches
217 Yards
3 TD’s
53.7 PPR Points🤯 pic.twitter.com/yRdO28mQx7

— Fantasy Sharks (@fantasysharks) February 20, 2020

A more effective way to look at player averages is to also look at their standard deviation. While it may sound like a lot, it will help you identify players that are prone to outliers. (If you want a simple way to plug in numbers and get a standard deviation, here’s a website that will do it for you). Player’s with higher standard deviations are more likely to have data points that are far away from their average. While you might want a player that is more likely to score significantly higher than their average, players with large outliers have positively skewed data and are more likely to score below their average.

Back to Will Fuller, the standard deviation for his points per game was 14.1. When we compare him to Odell Beckham Jr., who averaged just 0.4 points per game more than Will Fuller, we see that OBJ’s standard deviation is 6.0. At its simplest, this tells us that OBJ was more consistent with his points per game. (If you’re confused by standard deviation and what it means, here’s a simple run through and how it relates to distributions).

Final Thoughts

I wanted to start out this series with a simple concept and I hope the links help you have a firmer understanding of some of these basic statistics. I’ve barely scraped the surface of the issues of small sample size, and I look forward to diving deeper as I continue this series. Hopefully, this information helps you become a better fantasy football player and have a better grasp of statistics!

For questions, please reach out to Brian on Twitter: @Bpofsu

For more Undroppables analytics content, check out our RB Breakout Model, our 2020 Predictive Wide Receiver Model, and more on our analytics page.

References:

Button, K., Ioannidis, J., Mokrysz, C. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14, 365–376 (2013). https://doi.org/10.1038/nrn3475

Please support us on Patreon!