In a recent post at Arctic Ice Hockey, the indispensable Gabe Desjardins argued that we should move away from working on metrics for shot quality because there isn't much payoff. This has motivated me to write a follow up on an article I wrote a couple months ago on how luck vs skill influence shooting percentage based on sample size. In that article, I took two teams, one above average and one below average at shooting, and examined how likely the good team is to shoot at a higher percentage for a given numbers of shots. Here I will look at how much variation in shooting percentage is explained by skill and luck for different numbers of shots.
Methodology
My methodology is a sort of mirror image of what JLikens did in his article on the same subject and Vic Ferrari's imaginary dice rolling. JLikens assumed each team had the same real shooting percentage and ran simulations to see how much variation in results there would be after a season worth of even-strength shots. In my simulations I will create a distribution of shooting talent and see how much that skill explains variation in results for a given number of shots.
Here are the steps:
- Going by this more recent JLikens article in each of 10,000 simulations I created 30 teams by drawing a shooting percentage from Beta(263,2977), a distribution pretty close to that of actual even-strength team shooting skill in the NHL.
- All 30 teams take the same number of shots, which score a goal or not based on the probability given by the team's shooting skill.
- For each simulation, I calculate the R^2 between shooting percentage on those shots and the shooting talent of the teams. The average of these tells us how much variation in shooting percentage results is explained by shooting ability.
- The rest is luck.
Results
Here is a table with the results. The first column is the time period in question. The second is the number of shots each team took in the simulations. The third column is the percentage of variation in even-strength shooting-percentage results that is explained by the skill component - the average R^2 of all simulations. The last column is simply 100% minus that and represents the percentage of variation that is due to random chance, luck if you will.
| Time Period | Shots | Skill | Luck |
| Season to today | 250 | 9.9% | 90.1% |
| 1/4 season | 500 | 15.7% | 84.3% |
| half season | 1,000 | 25.1% | 74.9% |
| 1 season | 2,000 | 38.9% | 61.1% |
| 2 seasons | 4,000 | 55.4% | 44.6% |
| 3 seasons | 6,000 | 64.9% | 35.1% |
| 4 seasons | 8,000 | 70.9% | 29.1% |
| 5 seasons | 10,000 | 75.3% | 24.7% |
| 6 seasons | 12,000 | 78.4% | 21.6% |
| 7 seasons | 14,000 | 80.9% | 19.1% |
| 8 seasons | 16,000 | 82.9% | 17.1% |
| 9 seasons | 18,000 | 84.5% | 15.5% |
| 10 seasons | 20,000 | 86% | 14% |
Here's a graph:
Put in words, at this point in the season shooting results are 90% luck and 10% skill. This is likely an underestimate, as I'll discuss below. Over a whole season it goes to a little over 60% random chance. It takes about 140 games worth of shots for results to be 50/50.
Some thoughts
I'm making several assumptions that are not valid. The biggest and most obviously dubious is that shooting-percentage skill will be the same for every shot. In reality, if a team shoots at an 8.5% clip their top line will shoot higher, their fourth line lower, they'll do better against weaker opponents and worse against good goaltenders and so on. Injuries, trades, free agency and coaching changes are obviously a big issue as well. On a related note, I assume that each team takes the same number of shots. In practice, teams obviously take more or fewer shots than average over a given stretch. To make things more problematic, it is probably the case that teams that shoot more often tend to be better at making their shots. One thing all of these factors have in common is that they will increase the randomness factor. Think of the above estimates as lower bounds on how much luck explains variation in shooting percentage or an upper bound for how much skill matters.
Great read J. It's interesting in sports how much of a continum the skill vs luck debate has become. Idk if luck is really the right word, maybe variability, at any rate it's good to see these results put out over many seasons
ReplyDeleteThanks.
ReplyDeleteI think something like "random variation" is common, but I prefer to use the more everyday way of putting it even if it isn't quite right or is a bit loaded.
Another thing is that as Triumph has pointed out to me, a huge randomness factor isn't just the shots going in but the bounces that lead to the shots. An obvious and extreme example is the bounce off the stanchion for the Canucks last year. It's not strictly whether or not the shots go in.