As anyone who’s looked at evaluating goalies before knows it’s not a pleasant task. Even ignoring the inherent extreme variability in game to game or year to year statistics, a goaltender’s numbers can be heavily influenced by the 5 skaters in front of him. While when we look at forwards ordefencemen we can get around this issue by looking at WOWY or Relative numbers, we don’t have that same luxury with goalies. Even if we were to compare a goalie to how his team did when he had the night off our analysis is complicated by the fact that most teams tend to use only a small number of goalies each season. While we may be able to say that a team posted a better save percentage with their starter than their backup that doesn’t give us a good sense of whether the starter was great or the backup was terrible – we’re still stuck trying to isolate the team effect.
One method we can use to get an estimate of the team effect is their skill at shot prevention. Teams that are better defensively are likely to give up fewer shots, although somewhat paradoxically those shots are more likely to go in. But what if we looked at things from a different point of view: what if rather than focusing on a team’s ability to prevent shots in general we adjusted for who was taking the shots? After all, shots by defencemen tend to go in less than half as often as shots by forwards, so it stands to reason that if a team can prevent opposing forwards from putting the puck on net that their goalie would likely experience a higher save percentage because of that.
The first thing we should check before we get into this is whether the claim I just made (that teams can control whether shots against their net come from opposing forwards or defencemen) is actually true. The easiest way to verify whether something is more talent than luck is to check what the split half correlation is between even and odd games (in simpler terms we’re looking at whether the results we observe in the even games are a good predictor of the results in the odd games). There are two ways to look at this talent in our case and fortunately enough for us they both give similar results. The first way is to look at what percentage of a team’s even strength shots against come from forwards vs defencemen. In this case our split half correlation is 0.74, which is high enough to say that we’re likely on to something. The other stat that we can test is the total number of shots against per 20 minutes of even strength ice time that a team gives up to forwards and defencemen (fFA20 and dFA20, we’re using shot attempts here since I have the numbers handy but the results should hold generally for shots on goal). Under this method, our correlations are 0.80 and 0.72 for forwards and defencemen respectively, again good enough that we can feel confident that this is at least a team level skill*.
So now that we’ve established that teams can generally control where their opponents’ shots are coming from (positionally, at least) it naturally makes sense to adjust our goalie statistics (in this case 5v5 save percentage, which we’ll be using throughout the rest of this article) to take this fact into account. After all, if I post a 0.960 save percentage you’d likely be impressed, but if I told you that I only ever faced shots from the point your appreciation of my netminding skills would probably decline significantly. To do this we’ll start by looking at how many shots each goalie faced from forwards and defencemen, which we’ll call fSA and dSA respectively. After that we’ll calculate how many goals we’d expect our goalie to give up given their shots against distribution by position:
xGA = fSA * lgFSh% + dSA * lgDSh%
Where lgFSh% and lgDSh% are the league average forward and defencemen shooting percentage in each season, as given below:
Once we’ve calculated our expected goals against, we can turn this into an expected save percentage by simply dividing by the total shots each goalie faced:
xSv% = 1 – (xGA / SA)
We can then look at how each goalie actually performed against their expected value to come up with a relative team-adjusted (and in actuality also season-adjusted) save percentage:
adjSv% Rel = Sv% – xSv%
Which we can turn into adjusted save percentage by adding on the league average save percentage in the appropriate season (as shown in the table below):
adjSv% = adjSv% Rel + lgSv%
So now that we have our new stat defined, let’s take a look at the results. The table below has the standard 5v5 Sv% as well as the 5v5 adjSv% for each goalie who faced more than 500 5v5 shots over the past season. I’ve also included a column at the end which shows the delta between the traditional save percentage and our adjusted save percentage.
The way to interpret the delta is this: a positive delta means that a goalie’s adjusted save percentage is greater than his observed save percentage, or in other words that his defense cost him points by allowing more shots from forwards than we’d expect. As you can see, the deltas aren’t huge, but there are goalies for whom it does seem to make a difference in our evaluation. Reto Berra, for example, seems to have played behind the worst defence in the league, costing him roughly 3 points of save percentage over the course of the season (sorry, Flames fans). Over the 672 shots he took that works out to roughly 2 goals, or a third of a win that he was cost due to team defence alone.
On the other hand, both Jonathan Bernier and James Reimer’s save percentages seem to have been boosted by the Leafs’ defensive efforts (yes, you read that right). While the Leafs did give up a ton of shots, a greater percentage of those came off the sticks of opposing defencemen than we’d expect given the league-wide numbers.
While most of the deltas are relatively small, the raw differences don’t quite tell the whole story, as they don’t take into account the number of shots against a goalie faces. Rather than looking at raw Sv% and adjSv%, we can instead look at Goals Saved Above Average (GSAA) which attempts to measure how many extra goals a given netminder prevented over a league-average goalie:
GSAA = (Sv% – lgSv%) * SA
If we look at the delta in GSAA and adjGSAA, we see that the differences become more significant.
|Goalie||GSAA||Adj GSAA||GSAA Delta|
Looking at the delta column, the differences become a little clearer: both Ryan Miller and Reto Berra were cost about 35-40% of a win by their defences, while the defensive prowess of Colorado and Dallas likely added more than 1/3 of a win to each of Semyon Varlamov and Kari Lehtonen’s totals.
While these are still relatively small differences for many goalies, over the course of a career these can start to add up. Since 2008, the Rangers defensive play has saved Henrik Lundqvist roughly 8 goals, or about 1.5 per year. If we look solely at John Tortorella’s years behind the bench, the average delta works out to be more at almost 2 goals per year.
Expanding our sample back to 2008 also reveals larger discrepancies between GSAA and adjGSAA: both Jaro Halak (2010/11) and Craig Anderson (2009/10) were cost more than 3 goals by their team’s defensive play. Given that most goalies GSAA fall between -10 and +10, differences of even 1 goal can represent a fairly large difference in a goalie’s valuation. And it’s these differences in valuation that can shed light on some of the more difficult to measure aspects of the game, allowing us to get a better sense as to what’s within a goaltender’s control, and what we can lay at the feet of their teammates.
*Based on some initial analysis I’ve done this seems to be primarily a team/system level skill. If you look at it at the individual level and take into account how a players teammates did the dFA20 and fFA20 metrics show little year over year repeatability. But all that’s a topic for another post.