What’s a goalie worth these days? Determining replacement level goaltending

The concept of a replacement level player is one of the most important ideas in sports analytics, but one that unfortunately is used relatively infrequently in hockey. Replacement level is meant to represent the level of talent that’s widely available to any team at little or no cost. Comparing players against replacement level is important because it essentially tells us whether a player is worth keeping or not – anyone performing below replacement level should be discarded, since it’s likely that any free-agent or waiver wire pickup would perform better. It’s important to note that this is different than comparing against the average player – when we compare against the average we know whether a player is better or worse than most of the players eating up minutes throughout the league, but it doesn’t give us any indication of whether he’s worth keeping around.

For position players figuring out an appropriate replacement level is extremely difficult – there’s no broad agreement as to what individual statistics should be used, let alone any idea of how to determine who a replacement level player is. For goalies, however, we have a half-decent metric (Even Strength Save Percentage), that’s broadly accepted as being representative of individual talent, even if it is somewhat more variable in the short term than we’d like.

Calculating replacement level Even Strength Save Percentage isn’t a new concept at all: Scott Reynolds at Copper and Blue looked at goalies who faced fewer than 15 games during a season, while Bruce Peter at Habs Eye on the Prize broke playing time down further to classify goalies as Starters, Platoons, Back-ups and Call-Ups. Eric Tulsky took a different approach, looking at the aggregate save percentage of the goalies 61^st and lower by playing time to find replacement level.

The main issue with these methodologies, from my point of view, is that they tend to focus solely on playing time in their definition of replacement level. While this makes sense in a world where coaches and GMs make rational decisions regarding playing time, we know that there are definitely cases where that’s unlikely to be the case (Winnipeg, I’m looking in your direction).

The issue here is that there are certainly teams with starting goalies who they would likely ditch in a heartbeat for a slightly better option, while there are others who have backups that could start for most teams in the league. A better way to look at the problem is to go back to the definition of replacement level: goalies who are easily attainable for most teams at little to no cost. To find players who meet this definition I looked at all goalies whose cap hit was less than $1.5MM (which is somewhat arbitrary, admittedly, but should serve our purpose), or who were signed to entry level draft picks and were not picked in the first round. The first criteria looks for players who can be signed to cheap free agent deals, while the second criteria looks at who teams are able to draft without much opportunity cost.

So what exactly does a replacement level goalie look like? I pulled the data together from Rob Vollman’s excellent 2013-2014 Goalies Spreadsheet and calculated the aggregate Even Strength and All Situations replacement level Save Percentage for 2013-2014.

These numbers might look a bit high to a lot of people, but to me it highlights how tough it is to find a truly exceptional goalie in today’s NHL. By our definition, 59 goalies from Rob’s list qualify from as replacement level, while 38 are non-replacements.  If we limit our analysis to goalies who faced at least 500 even strength shots last year, 35 of them posted raw ESSV% above replacement level, while 15 were at or below.

Having put together our definition of replacement level, we can now look at how much each player was worth to their team in the past season. The table below lists each goalie who faced at least 500 shots last year, along with their Adjusted Save Percentage, Adjust Goals Saved Above Replacement, and Adjusted Wins Above Replacement (ignoring everything I wrote about in my last post I’ve used the generally accepted 6 goals/win to calculate this). Adjusted Save Percentage is meant to take into account each teams variation in their “shots against mix”, that is their propensity to give up shots to opposing forwards (who shoot well) or defencemen (not so much).

At the top of the table, Tuukka Rask is very clearly earning his new contract-at 7MM per season for the top goalie in the league, you have to feel that the Bruins are getting their money’s worth. Semyon Varlamov and Carey Price also posted strong seasons, although a lot of their worth is driven by their teams’ inability to prevent shots against. On a team with an average number of shots against, they’d be knocked down a few notches on the list.

On the other side of things, you really have to wonder what’s going on with the Jets, who have recommitted to Ondrej Pavelec in spite of the fact that he cost them 4 pts. over the past year with his play. Pavelec has only been above replacement twice in the past 5 years, and only once was he worth more than one win. Devan Dubnyk on the other hand is an interesting case: he’s been above replacement level every season but last year, but seems doomed to sit behind the comparable but more expensive Mike Smith in Phoenix Arizona next year. And then there’s Marty Brodeur: while I’m likely in the minority of the fancy stats crowd who believe that Brodeur is underrated (the Devils limited shots against likely suppresses his save percentage numbers), I do feel that he’s past his prime, and anyone looking for a goalie for next year would be wise to take a pass.

While calculating replacement level helps us get a better sense as to what a goalie has done, it still has its limitations: goaltending results are extremely variable season-to-season, so even if we have a solid baseline to compare against we can’t necessarily know whether a goalie would be expected to repeat his performance the next year. Nonetheless, it does give us a better ability to decompose a team’s win total into its principal components, which will in turn translate into better valuations for the skaters on each team.