Yesterday, I posted on NHL shootouts, and the response was great. A few follow-up points.
1) I was too vague late in the article when I tried to weave in an effect size of goaltender differences on a team’s win percentage. Here’s what I wrote:
Using these numbers, and assuming a team goes to the league average 13 shootouts in a season, the difference between a top goalie and a poor goalie is about 3 or 4 points alone, simply due to their shootout performances.
I’ll clarify here.
What I should have said is that their past performances have been worth 3-4 points a year, based upon their shootout save percentages. This doesn’t necessarily hold for future performance, though. Further, I was using the observed goalie win percentages to calculate those numbers, which probably wasn’t the best way to go about things.
Coincidentally, Tango Tiger just posted on this same topic, and posits that a top goaltender can expect about a 5-6% save percentage increase moving forward (accounting for some future regression), over an average one.
2) Because I was vague yesterday, let’s try and clarify a better effect size today. For example, I’ll try and determine the differences between a top goalie (say, 1 standard deviation above the mean moving forward, or a 5.5% higher save percentage), and both a league average one (0 sd’s) and a relatively poor one (1 sd below the mean).
In other words, let’s compare the shootout team winning percentages of three goalies with save percentages of 72.7, 67.2, and 61.7%, respectively. For this, I’ll naively make a few unverifiable assumptions. First, I assume that each team is facing a league average offensive in every shootout. Second, I’ll assume that every shot is independent of the one before it (and, moreover, that shooters or goalies don’t get better or worse as the shootout drags on).
In any case, under those assumptions, I simulated 1,000 shootouts, and here’s what I found:
Top goalie (+1 sd) vs. Average goalie (0 sd): Top goalie wins about 58% of the time
Top goalie (+1 sd) vs. Bottom goalie (-1 sd): Top goalie wins about 66% of the time
Thus, under our assumptions, a top goalie will beat a bottom goalie in the shootout about twice as often as he’ll lose.
3) Unless Fleury (top) is facing Backstrom (bottom), however, the most meaningful number is likely how the top goalie would perform against a league average opponent. At a 58% clip, this suggests that the top goalie, on average, can be expected to win one or two more shootouts per year than the average goalie, all else being equal. This isn’t a large effect size, but it’s certainly better than a coin flip.
4) Someone found an interesting trait shared by our top two shootout goalies.
While this is probably a stretch, perhaps it is possible that the Penguins and Rangers have been over-seeded in recent playoff competition, in part because their goaltenders inflated their team’s playoff positioning with outstanding shootout performances. Further, because shootouts are few and far between in the postseason, this advantage disappears come playoffs.
5) I found a few more links to be useful reads.
Clown Hypothesis, using a larger pool of goalies, looked at net-minder performance a few months back (here) and found a few more names to add to the list of exceedingly good or poor goalies.
Back in 2012, Eric T followed up Michael Schuckers’ piece, looking at several aspects of shootout performance.
https://twitter.com/BSH_EricT/status/443913636382138368
6) Lastly, I received some excellent ideas, questions, and suggestions, all within 24 hours. Even when some of these comments are critiques, I still think this is great. From a personal perspective, I contrast this to submitting research papers in academia, which yields a review process that lasts several months, only until a very small subset (usually 2 or 3) of reviewers and experts give you their opinion.
Posting on hockey, I got more than a dozen comments instantaneously.
Advantage, blogging.
StatsbyLopez 
I think there might be a problem with your simulation. If I follow correctly, you are using observed success rates in place of true success rates. For example, a goalie might have a 65% win rate in shootouts one year, but that doesn’t mean his underlying true probability of winning a shootout is 65%. It’s much more likely that his true win rate is something much closer to 50%, but he was a bit lucky that season. The same concept applies to an entire career.
So in your simulation, you’ll need to regress your win % by some considerable amount. I think you’ll find that a +1 SD goalie beats a -1 SD goalie much less often than you have found above.
You can figure out how much to regress those rates by either of two methods. You can build a simulation and vary the distribution of “true” win% until you get the same distribution of “observed” win% we actually see in the NHL. Or you can directly compute the variance, as I outlined in your post from yesterday.
Hi Ted,
I used the regressed rates that Tom Tango used in his blog post, and not the past rates. This takes the top goalies from 2-3 SD’s above average to 1 SD above average.
http://tangotiger.com/index.php/site/comments/most-talented-shootout-nhl-goalies
I misread your post slightly. I see you are using save rate to simulate shootout outcomes, rather than using shootout outcomes to simulate shootouts. My suggestion above applies just the same, though–You’ll need to regress save rates toward the league mean before using them in the sim.
0) Cool stuff.
1) I think it’s unnecessary to do this via simulation—given the assumptions you make, you can compute the exact probability using some only slightly messy binomial calculations, which I did over on my blog (and they match yours fairly well, actually). (http://clownhypothesis.com/2014/03/15/valuing-goalie-shootout-performance-again/)
2) How does a top goalie get to two (expected) wins above average? Unless I’m missing something, an 8% difference means 0.08 expected extra wins above average, which means that we need 12-13 shootouts for a goalie to amass 1 expected win. Just looking at the NHL.com stats, it looks like there’s only about 1–5 goalies playing in 12 or more shootouts per year. I think the “one or two” might be an overstatement.