Evaluating goaltenders and goaltending prospects

For the last few years, I have approached my prospect rankings very conservatively on how highly I rank goaltenders pre-draft and post-draft. It has been several years since I ranked a goaltender in a top 60 in an NHL Draft ranking. In my Top 100 NHL Prospect ranking , I had only three goalies ranked: Tampa Bay’s Andrei Vasilevski at 79, Ottawa’s Robin Lehner at 92, and Anaheim’s John Gibson at 97. I don’t expect everyone to fully agree with this line of thinking. Scouts think I’m too harsh on goalies; stat analysts think I’m too lenient.

Here is my explanation for why goalie prospects should be ranked conservatively. However, we can’t get to the goalie prospect part until we understand how valuable goaltenders really are in the NHL, which is the end goal.

NHL goalie value

The only goaltending stat referenced in this piece will be save percentage. This is because stat analysts have found that team level shot quality is not a significant skill. This is not to say a breakaway is as dangerous as a blue line wrist shot, but while single season shot quality can create very large effects on results, it has not been found to be a persistent team skill. The observed shot quality by teams in single seasons is mostly just randomness as opposed to the effects of coaching tactics or personnel. Teams have not shown the skill to replicate shooting percentage, but they have for shot differential and goaltenders have shown a minor skill to replicate save percentage. For this reason, most respected stat analysts only use save percentage, usually at even strength, as the one main result stat to evaluate NHL goalies by. Some analysts have shown shot quality may be a minor team skill , though. The reasoning for this will be expanded on in a future article.

Goaltenders can be tricky to evaluate because they are frequently the most impactful players in single games or seasons for a team. Whether a goaltender stops 30 or 34 of 35 shots against swings the result of a game dramatically. Using the Goals Versus Threshold (GVT) metric, an all-encompassing value stat like baseball’s Wins Above Replacement (created by Tom Awad of Hockey Prospectus), the top five players in the NHL last year were goaltenders: Sergei Bobrovsky, Henrik Lundqvist, Tuukka Rask, Antti Niemi, and Craig Anderson. In fact, nine of the top 20 players by GVT were goaltenders.

However, it is extremely difficult to project goaltender performance, which thus lowers their value. Using the last two 82-game seasons, 13 of the top-30 forwards in 2010-11 repeated as top-30 scorers in 2011-12. Meanwhile, three of the top 10 goaltenders with regular playing time (30 appearances) who finished in the top 10 in save percentage in 2010-11 finished in that range the next season; they were Henrik Lundqvist, Tim Thomas, and Pekka Rinne. While that is a roughly comparable repeating rate to forwards in this example, considering there are 360 “top-six forwards”and 30 starting goaltender jobs, it is very indicative of how hard it is for goalies to repeat great performances.

A fair number of NHL fans know that goaltenders are very hard to evaluate on some level. Most have probably observed their favorite netminder fail to produce a great season after impressing so many the season before or seen a lesser-name goaltender simply play amazing out of nowhere (we’re all looking at you, Brian Elliott). Gabriel Desjardins, who has done analysis for several front offices, says you need four full years of performance before a goaltender’s save percentage become reliable enough to evaluate due to how random save percentage is. Awad showed that in one season with at least 40 games played, only 33% of a goaltender’s save percentage is observable skill, while 66% is due to random chance. Over three seasons, that latter number only comes down to 50%. Even when you get years worth of data, outside of the very good and very poor goaltenders, three quarters of goaltenders in the NHL have performance that is “almost indistinguishable” when you get a reliable amount of years. Detroit’s GM Ken Holland echoed this in the book Behind the Moves when he said, “If you told me I had one of the top five best goalies in the world, I’ll put money in the goal, but I believe the difference between the 10th and 17th best goalie in the world isn’t much.” Over the last four seasons and including this season, Jimmy Howard is seventh in save percentage among regular goaltenders (.919).

The reason for the small difference between most goaltenders is hard to pin down, but my belief is it is likely due to the mechanical nature of modern goaltenders with an emphasis on technique. It may also be due to the increased ability to groom amateur goaltenders and the impact of a global game, combined with the low number of jobs in the NHL.

To make matters worse, some rinks count shots differently, and that influences save percentage. Awad showed the difference between arenas a few years ago. While Martin Brodeur’s rink may have undercounted his save percentage, Nashville, which consistently seemed to get great goaltending, always generously awarded saves. It is why some analysts suggest using road save percentage only; unfortunately, this cuts sample size in half, hurting accuracy that way.

Therefore, barring extremely good or extremely bad seasons, it is almost impossible to dissect anything meaningful from single season goalie numbers with current statistics.

Goaltender prospects

If the stats are not reliable, one may say to just trust the scouts. However, amateur scouts have historically fared just as poorly as the NHL statistics in terms of evaluating goaltenders. The median draft slot for the top-20 goalies drafted since 1990 is 107th overall. For forwards, it is 3.5 and for defensemen it is 44.5. The Henrik Lundqvists, Pekka Rinnes, and Tomas Vukouns of the world have contributed to that, getting drafted much later on. Some NHL teams have acknowledged the difficulty in scouting this area and have hired scouts to evaluate just the goaltending position. When I asked one Eastern Conference scout about a goaltender draft prospect the other day, he responded, “I don’t scout goalies anymore, our goalie guy takes care of that.” The difference in analysis between how regular scouts and goalie scouts or goalie coaches evaluate can sometimes be very different as well. It is almost like they are talking different languages. The former (not to generalize the knowledgeable scouts I talk to) tend to focus on athleticism, a goaltender’s reads and compete level, while the latter can pinpoint the most minute detail in a player’s mechanics.

Another issue: Goaltender is a very tough position to learn and takes time to master at the pro level. Only two netminders under age 23 played 10 games or more last season, Jake Allen and Robin Lehner. Meanwhile, there were 100 players at forward and on defense under age 23 who suited up for at least 10 games. This is due to a combination of how tough it is to play a position, the adjustments players make at the AHL level that take time, and the lack of available jobs in the NHL. “It’s a really hard position to learn, which is why it can take so long” said one NHL executive.

What about tracking goaltender statistics in the amateur ranks? Well, if stats for goalies in the NHL are very problematic, wait until you hear about major junior. Matt Pfeffer, statistical analyst for the OHL’s Ottawa 67’s, told HP that 17 of 20 OHL rinks overcount shots. Over the last few years, one OHL team’s rink has counted 30% more shots for their team at home versus how they fared on the road. If 70 game seasons in the NHL are questionable to determine statistical reliability for goaltenders, 50 games of very sketchy data accuracy in the CHL can make goalie stats almost useless. Even if you assumed all rinks overcounted, having to adjust for scorers makes it very challenging even if you just statistically evaluate goaltenders relative to their CHL league. Additionally, you only typically get one CHL season where a goalie has started before he needs to be drafted. In rare cases like Montreal prospect Zach Fucale, you can get two.

One NHL stat analyst who has done work for an NHL team also felt shot quality in junior was more apparent than in the NHL, which is a reasonable theory. It is harder to prove, though, due to rapid development curves for junior players, high roster turnovers, and shorter seasons.

Bringing it all together

The quick summary of everything I have just said is as follows. Goaltenders in single games can swing its dynamics unlike any other position and can be among the most valuable players in the league. Save percentage is the only stat that is really useful for NHL goaltending performance, and it is really unreliable. You need years worth of save percentage data for it to be a useful stat for determining the true talent of a goalie, and you will still be wondering if it is even accurately tracked. Once you have waited years and found out how good a goalie really is, unless you have one of the very best or very worst starting goaltenders, chances are the other 20-25 goaltenders are not all that different skillwise from each other.

When it comes to prospects, scouts have not been able to identify the top amateur talents during their draft season. Even when they were to identify them correctly, even the best goaltending prospects take many years before they can get to the NHL and establish themselves in a starting role. Stats outside the NHL do not have a ton of usefulness for 17-18 year old goaltenders.

Even with all this said, and you may be surprised to hear this, I still think it is worth investing in goaltending prospects, but at the right cost. If a team identifies a player as a top-end talent between the pipes, it may be worth a top-60 pick, although chances are those players will be picked higher. Also, if you identify a raw but talented goalie, it is worth using a late round pick, especially for depth chart reasons. The college and Euro free agent market can be appealing for goalie pickups as well.

As a hockey community, we don’t fully understand the goaltending position as well as we understand the skaters. Further advancements in puck tracking and more in-depth goaltending metrics at the NHL level may help scouts identify the most important traits for predicting goaltending success. Scouts have shown the ability to identify elite talents like Roberto Luongo and Tuukka Rask, but obviously need a better hit rate with goaltenders in the first round or even in the top 45. A development such as knowing if goalies have a predictive element to the type of shots they save, or by the location of the net targeted. Figuring out if something (and this is purely out of thin air) like high saves are more predictive of future success while perhaps screened shots are a form of shot even better goaltenders cannot stop much better than the rest. Digging deeper into the result stats to describe the “why” is what developed the Corsi line of thinking in the blogosphere that has now trickled down to NHL front offices. Further advancements in goaltending along similar lines could help dramatically. There are also rudimentary methods available to help analyze goalies statistically.

I feel a better hit rate in the draft with goalies is possible, but until we as a hockey community find a way to understand goaltenders better and have a better grasp of what predicts success, I have a hard time ranking a goaltender highly in an NHL Draft ranking barring some moderate confidence that he will be great.

PronmanCorey Pronman is a long-time author of Hockey Prospectus, well known for his outstanding prospect work, and having contributed on all four annuals. And as of this season, he is ESPN’s NHL Draft and Prospects analyst. He provides analysis on the top draft-eligible players, prospects drafted by NHL teams and all other relevant prospect information.

Follow Corey on Twitter at @coreypronman.

Leave a Reply

Your email address will not be published. Required fields are marked *