NHL shootouts aren’t random. So let’s stop calling them that.

(for Part II, click here)

The National Hockey League has once again made headlines for tweaking its standards of playoff qualification, this time deciding it would look into into modifying the league’s shootout system. In the current structure, a shootout concludes any 5-minute overtime session ending without a goal, with the winning team earning an extra point towards season standings

One of the major factors in the NHL’s thinking, as it turns out, is that shootouts are too random. Just today, a Toronto columnist called the shootout a coin flip.

Statisticians, including ones I have a lot of respect for, have frequently enforced this idea. In an old post on SB Nation, St. Lawrence associate professor Michael Schuckers walks through several analyses to show that the distribution of player and team performance, by and large, is no different than what we see due to chance. I really liked Schuckers’ article, and hopefully this post builds on his work to present another perspective.

Before exploring some data, let’s get a couple of important things out of the way.

First, I hate shootouts. They are the equivalent of ending an NBA game in a game of HORSE. Deciding a game by skills competitions is silly.

Second, I hate the NHL’s point system. More specifically, I hate how shootouts have a large role in dictating playoff positioning within the point system. Roughly 15% of NHL games are won on a shootout, and those extra points play critical roles in which teams qualify for the playoffs.

So while I agree the NHL’s system needs an overhaul, let’s stop calling shootouts random.

Here’s why.

1) Only a team’s best players take its shootout attempts. 

In a shootout, coaches have their choice of players to take each shot. Shootouts can be decided in as few as three attempts per team, and it is rare for shootouts to last longer than 5 or 6 rounds. Unlike the Olympics, where the USA could keep trotting out TJ Oshie against Russia, NHL coaches are not allowed to use their players more than once.

But which players do the coaches use?

With near uniformity, they choose their forwards. In fact, of active players, only one of the top 70 in shootout attempts is a defensemen (Kris Letang is 42). In other words, coaches do not randomly employ players to take shootouts, and if they did, they are well aware that they’d probably lose much more often. Operating under the assumption that forwards are better at shootouts than defensemen, immediately we recognize that something is inherently not random about shootouts.

Further, if we agree that forwards are better than defensemen, is it not feasible that certain forwards are better than other ones? While these within-forward differences might not be large, it certainly plausible that they do exist. Moreover, part of the reason it might be difficult to distinguish good shootout forwards from bad ones is that bad ones stay on the bench!

Further, even after ten years of the league using a shootout, we might not have enough data to detect offensive player or team-specific shootout effects.

The same can’t be said of goalies.

2) Goalie performance is not random. 

I extracted all shootout data for active goalies from NHL.com, restricting myself to the 41 goalies with at least 50 shots faced in his career.  The average goalie in the NHL during this time period stopped 67.2% of shootout attempts.

Here’s a plot of the 41 goalies in my study, along with a confidence interval for shootout save percentage.

While Nicklas Backstrom (save percentage, 56%) lies well below the league average, Henrik Lundqvist (75%) and Marc-Andre Fleury (78%) perform notably better.

Of course, these fluctuations could simply be due to chance. With 41 goalies, we’d expect a few to have save percentage confidence intervals falling outside the league average.

Thus, our critical question is: Do the observed distribution of goalie save percentage deviate from what would have occurred if every goalie’s save percentage was truly 67.2%?

To answer this question, I used the total number of each goalie’s shootout save attempts (minimum of 50, and Lundqvist had the most with 150) and the Binomial distribution to simulate the number of shootout saves that each goalie would have made with a NHL average shootout save percentage.

Here’s an example of the same graph as above, only with randomly distributed save percentages (and random names, since every goalie is equal in my simulation)

Of course, one simulation by itself tells us next to nothing.

So I did this same thing 10,000 times, and at each simulation, I extracted each players critical value: i.e., the number of standard deviations his simulated save percentage lies away from the mean. A standard deviation of -1.00, for example, would indicate that the goalie’s simulated save percentage was 5.5% below the league average.

These simulations represent what would have occurred if these goalies had been observed 10,000 times at the league average save percentage. At each run, I sorted the critical values, and averaged across simulations to get a sense of how much deviation from 67.2% we would expect to see by chance.

Here’s a plot of the observed test critical values (x-axis) versus the simulated critical values (y-axis), along with a line (y = x) which represents what would have occurred if the observed distribution equaled the simulated distribution.

In the figure above, it is clear that there are goalies who are better and worse than we would expect to have seen by chance. The observed test statistics deviate from the quantiles which we would have expected if every goalie was truly equal. In other words, it is rare to have the league’s worst goalie be as bad at shootouts as Backstrom has been, and to have the league’s best goalies be as talented as Fleury and Lundqvist.

To get a sense of just how much the NHL goalies deviate from their average, I summed the absolute value of the test statistics at each simulated draw (let’s call this value SumZ). High SumZ will indicate simulations in which several critical values fell far away from 0, while low SumZ will indicate one where most critical values fell close to 0.

Here’s a histogram of the simulated SumZ values, along with a point representing the observed SumZ of 46.9.

In only 11 of the 10,000 simulations were NHL goalies as random as the variability which has been observed. This extremely low probability provides strong evidence that goalie performance is not due to chance.

Of course, if goalie performance isn’t random, then shootouts aren’t, either. 

To conclude, while there are certain aspects of a shootout – specifically, that offensive performance might be highly variable from year to year, and that the players we expect to be great at shootouts might be terrible at them – this post provides two examples of shootout aspects which do not appear to be due to chance.

For what its worth, I looked at the win percentages of the goalies in our sample. Fleury and Lundvquist, our top two netminders, have combined to to win 91 of 142 of their games (win percentage, 64%), while Backstrom, our worst goalie as judged by save percentage, has won just 22 of 55 (40%).

Using these numbers, and assuming a team goes to the league average 13 shootouts in a season, the difference between a top goalie and a poor goalie is about 3 or 4 points alone, simply due to their shootout performances.

(for Part II, click here)