Yesterday, I posted on NHL shootouts, and the response was great. A few follow-up points.
1) I was too vague late in the article when I tried to weave in an effect size of goaltender differences on a team’s win percentage. Here’s what I wrote:
Using these numbers, and assuming a team goes to the league average 13 shootouts in a season, the difference between a top goalie and a poor goalie is about 3 or 4 points alone, simply due to their shootout performances.
I’ll clarify here.
What I should have said is that their past performances have been worth 3-4 points a year, based upon their shootout save percentages. This doesn’t necessarily hold for future performance, though. Further, I was using the observed goalie win percentages to calculate those numbers, which probably wasn’t the best way to go about things.
Coincidentally, Tango Tiger just posted on this same topic, and posits that a top goaltender can expect about a 5-6% save percentage increase moving forward (accounting for some future regression), over an average one.
2) Because I was vague yesterday, let’s try and clarify a better effect size today. For example, I’ll try and determine the differences between a top goalie (say, 1 standard deviation above the mean moving forward, or a 5.5% higher save percentage), and both a league average one (0 sd’s) and a relatively poor one (1 sd below the mean).
In other words, let’s compare the shootout team winning percentages of three goalies with save percentages of 72.7, 67.2, and 61.7%, respectively. For this, I’ll naively make a few unverifiable assumptions. First, I assume that each team is facing a league average offensive in every shootout. Second, I’ll assume that every shot is independent of the one before it (and, moreover, that shooters or goalies don’t get better or worse as the shootout drags on).
In any case, under those assumptions, I simulated 1,000 shootouts, and here’s what I found:
Top goalie (+1 sd) vs. Average goalie (0 sd): Top goalie wins about 58% of the time
Top goalie (+1 sd) vs. Bottom goalie (-1 sd): Top goalie wins about 66% of the time
Thus, under our assumptions, a top goalie will beat a bottom goalie in the shootout about twice as often as he’ll lose.
3) Unless Fleury (top) is facing Backstrom (bottom), however, the most meaningful number is likely how the top goalie would perform against a league average opponent. At a 58% clip, this suggests that the top goalie, on average, can be expected to win one or two more shootouts per year than the average goalie, all else being equal. This isn’t a large effect size, but it’s certainly better than a coin flip.
4) Someone found an interesting trait shared by our top two shootout goalies.
While this is probably a stretch, perhaps it is possible that the Penguins and Rangers have been over-seeded in recent playoff competition, in part because their goaltenders inflated their team’s playoff positioning with outstanding shootout performances. Further, because shootouts are few and far between in the postseason, this advantage disappears come playoffs.
5) I found a few more links to be useful reads.
Clown Hypothesis, using a larger pool of goalies, looked at net-minder performance a few months back (here) and found a few more names to add to the list of exceedingly good or poor goalies.
Back in 2012, Eric T followed up Michael Schuckers’ piece, looking at several aspects of shootout performance.
6) Lastly, I received some excellent ideas, questions, and suggestions, all within 24 hours. Even when some of these comments are critiques, I still think this is great. From a personal perspective, I contrast this to submitting research papers in academia, which yields a review process that lasts several months, only until a very small subset (usually 2 or 3) of reviewers and experts give you their opinion.
Posting on hockey, I got more than a dozen comments instantaneously.