Much of the modern day advances in statistical analysis in hockey has been based around shot-based analysis. Be it Corsi, Fenwick or Delta, the shot-based metrics provide terrific benefits over standard plus-minus by upping the sample size of data exponentially, taking the goaltender out of the equation and analyzing the game of hockey at a skill that is more persistent than goal-based data. However despite all its benefits, the stats still don't isolate player performance, rather just providing data of what happened when the player was on and off the ice.
The solution to this is more detailed score-keeping, which several NHL teams do on a regular basis for their own organizations. David Staples from the Edmonton Journal's Cult of Hockey has also been doing a more advanced style of scorekeeping for the Oilers the last few years. I interviewed him about the process and the results:.
Corey Pronman: Could you describe your scorekeeping method and the data you track?
David Staples: I PVR every Edmonton Oilers game. I watch the game looking for scoring chances, as they are the most significant event of the game. Scoring chances lead to goals. The more chances over time, the more you're going to score, and the more you score the more you win. Creating scoring chances is the single most important activity related to winning in the NHL, other than goal scoring of course, but whether a goal or scored or not on any given chance often comes down to luck.
So if you want to know the true underlying talent of players, it is best to see how many scoring chances they have individually contributed for or a mistake on a chance against. I look at a tight definition of a scoring chance and I go by what Buffalo Sabres coach Jim Corsi told me. He only counts hard shots from the slot area which I define as a shot between the faceoff dots and from the top of the circles in towards the net. Billy Moore, the former assistant coach of the Oilers, calls a scoring chance as a hard shot from the point when the goalie is screened, which I also count as a scoring chance if it's a very good shot with a screened goalie.
So in the end, I'm left with an even-strength plus-minus for a player based on how many chances he's created and how many mistakes he's allowed on chances against which is the same thing I've done for goals for the last three years. With the scoring chances data, it's helped me see which players have been lucky or unlucky based on how many chances they've created or prevented, and whether or not their chances go in at a higher percentage and their mistakes end up as goals against or not.
I also count, as a side project, what I call hard chances at the net. When there is a scoring chance, if there's a screen, a hard charge at the net, jamming the net or a deflecting of the puck at the net, I count that as a hard scoring chance at the net. So it gives me an idea of which players play a certain kind of game, be it a tough inside game or an outside game. Ales Hemsky has very few hard chances at the net, but is an effective hockey player, whereas Dustin Penner has many hard chances at the net.
On chances against, what I'm looking for are the mistakes and what they tend to overwhelmingly be is turnovers, lost battles and missed assignments. Bad line changes, screening the goalie and deflecting it against your own goalie are also mistakes you will sometimes see on chances and goals against. I also go back to when there was an initial mistake or a good play and the possession changes and from then on (within 10-12 seconds on average) to the time of the goal. I count all the good and bad plays from that point on.
Pronman: Do you feel that covering a team as poor as the Oilers have been the last little while has influenced your data at all?
Staples: In 2008-09, the Oilers were an average team at ES but with terrible special teams. At the same time, the Oilers have had terrible players, but they've had some quality players like Penner, Hemsky and Whitney so I've been able to see what good NHL players do and how they contribute to a team.
Pronman: What are some general observations or findings that you have gotten from this method?
Staples: If you're going by regular box score stats, they undervalue two kinds of players: the players who go hard to the net and the defensemen who initiate the attack out of their own zone as well as centers in that role. Applying a team plus-minus stat is very misleading in many cases. They assign false positives and negatives at least 33% of the time and up to 50% of the time. So using team plus-minus to evaluate individuals is essentially useless. Unless you actually see the players, by using plus-minus you have little idea who is driving the bus.
Pronman: About what percentage of plays that contributed to goals were created by physical plays that wouldn't be recorded with box car stats?
Staples: About 20% of the plays that contribute to goals in the official stats are based on physical plays like a won battle or a hard play at the net. On offensive stats, the stats are good to okay, but when it comes to evaluating a player's physical play, defensive game and his overall game, they just don't get the job done.
Pronman: Were there ever cases where you felt a player was misrepresented based on your evaluation in comparison to their plus-minus?
Staples:Some players are responsible for a high percentage of the goals they're against for, yet a strong defensive player is only responsible for about 20-40% of the goals they're on the ice for. Shawn Horcoff is a great defensive player, yet he had a horrible plus-minus because he was on the ice with rancid Patrick O'Sullivan and based on plus-minus you couldn't tell who was worse. But with my measurement you could.
Pronman: What about when comparing players by your evaluation to their Corsi?
Staples: Corsi is a better stat than regular plus-minus but it unfortunately has the same weaknesses of false positives and negatives. It will give you a misleading impression of the players. I looked at how Corsi ranked players at certain positions and compared it to mine. If you're trying to eliminate the worst two-way winger from your team, Corsi told you a different player than my scoring indicated. Hockey decisions often come to one player or another, so with the fact Corsi and my scoring you get different answers, and if you went by Corsi you'd keep the wrong guy. It gets it wrong, especially I'm when using my scoring chance data for individuals, I find it's much more precise than using Corsi. People should move on to a better practice than Corsi plus-minus, which is studying scoring chances.
Pronman: Was this still apparent even if you looked at Corsi adjusting for performance-altering factors like quality of competition, linemates and offensive zone starts?
Staples: You need to adjust all stats for quality of competition. With Corsi, you also need a huge focus on zone starts, otherwise that data is really out of whack. But, really, the future of rating hockey players isn't Corsi plus-minus; it's just been used because it's widely available. What's really needed is scoring chance data, where people who know a lot about the game also ascertain which players help to cause chances for and make mistakes on chances against.
Pronman: What is your response to those who believe using shot-based statistics are an effective tool for player evaluation?
Staples: Everyone has access to Corsi numbers for all teams and they base their entire research and evaluation of players off it and I feel when someone comes along and says: "You're wrong and my way is better," it's not welcome news. No one wants to hear that their favorite method isn't a fair and accurate way to evaluate players. But for what it's worth, that's how I see it with Corsi. Even folks who believe in it are moving on to scoring chance plus-minus, but they run into the same old problem when they don't break down the game recording to ascertain individual responsibility. They're left using a team stat to try and rate an individual player, but that method is utterly riddled with false negatives and false positives.
Now the one down side to what I do in rating scoring chances is that it's subjective. Someone has to determine who made significant contributions to the chance, or mistakes on the chance against, when the scoring chance sequence actually started, whether or not the shot was an actual scoring chance. But the fact is fans who have watched the game for decades can do this work. And as I've found out, NHL teams have done this same work at least since the 1990s. Billy Moore of the Oilers has done it. The Rangers have done it.
I was relieved when I found this out. I don't want to be some stats guy trailblazer. I want to use methods that make sense, that are tried and tested, that really get to the heart of fairly and accurately rating the performance of NHL players. And I do think that's what studying individual scoring chances does.
Pronman: Based on your tracking of scoring chances, what aspect of a player's game do you feel is most valuable and least valuable from traits like skating, puck skills, passing, shot, physical play and hockey sense?
Staples: I'll take a team of guys with superior hockey sense every day of the week. The guys who can read the game know how goals are scored on offense, by screening the goalie and taking the puck hard to the net, and they know that on defense, it's all about keeping your body between the attacker and the net, and it's all about stopping that deadly pass to the open shooter in the slot. I like smart players. On the team I focus on, the Oilers, Shawn Horcoff, Dustin Penner and Jordan Eberle are the three smartest players when it comes to hockey sense. That's not to put down Taylor Hall, who is also a smart player, but those other three make up for some talent deficits with their smart heads for hockey.
Two experts weigh in
I also asked Tom Awad and Gabe Desjardins, two of the industry's leaders in hockey analysis, on which method they would prefer between scoring chance data and shot-based data.
Tom Awad:There's almost no question that scoring chance data would be preferable. As it is now, with Corsi or Delta, we must assume that all players on the ice are equally responsible for every event, which is certainly false. The only issue would be in the definition: oftentimes a forward can be
responsible for a scoring chance against through poor backchecking even though he
wasn't involved in the defensive zone play. What's the contribution of the guy screening
the goalie versus the guys setting up the passes? And so on. While there is necessarily a certain amount of subjectivity, something along those lines may turn out to be an improvement on the metrics we are using now.
Gabe Desjardins:It doesn't really matter. They all converge to roughly the same thing, and they have enough uncertainty in them that any one of them could give the best answer for a given player.
I don't think we can conclude which method is better because as it stands now we don't have enough scoring chance data to fairly compare it to shot-based data. Gabe mentions how they converge roughly to the same thing, which makes sense when past analysis has proven that shot quality isn't really a persistent talent, so shots could be seen as a proxy for scoring chances. The only concern I have is does shot-based data get it right on the individual level without isolating the player, and based on what David said in relation to his study, he saw some errors in that regard.
It should also be noted, based on several conversations I've had with NHL head coaches, that numerous NHL teams track individual contributions to scoring chances and errors against for their own players on an every game basis. As David said, this is not some sort of new concept. Last summer, Vic Ferrari showed how the idea for Corsi was being utilized decades ago, so the fact that hockey people use statistical methods that aren't in the mainstream isn't really a surprise. Gabe has also mentioned before howteams tracking chances isn't exactly news.
One of the more significant findings mentioned by David was the significance of a player's physical game. We often hear the size debate in hockey circles but nobody can ever put a value on it. Correlating data with height and weight helps, but that's not the core of the issue. As I said over at Behind the Net Hockey, size isn't a perfect correlation to physical dominance as big players can lack and small players can excel in that area, but it helps a lot! A critical aspect of evaluating a player's physical game though is that physicality is not always measured in Goals or Assists, as David found out. Per David's study, having 20% of your contributions come from outside those boundaries is a major gap in modern day statistics and while we have more advanced ways to evaluate players in the NHL, those deficiencies really trickle down to the AHL, NCAA and CHL levels and make prospect analysis that much harder. It should also be noted that since physically-gifted players can go outside of Goals and Assists more often for their contributions, it makes comparing them to lesser physical players by those metrics that much more difficult. It shouldn't be a surprise that smaller, skilled players often succeed on the power play and rack up counting numbers when they have more room to work with and that better physical players will be good even-strength contributors when the situation calls for them to go to the corners and sideboards and dig out pucks and cycle.
At the end of the day, I can't really make any firm statement as to which method is better or not, because the data simply isn't there as of now. Plenty of bloggers have started tracking on-ice scoring chances, and hopefully one day they'll move onto isolated contributions. Until then, it's definitely something to keep in the back of your mind.
If you're interested in seeing more about the method, head over to David's blog for more detailed results with the data broken down into categories. The lost battles results are pretty significant and again show how important the physical game is especially for defensemen, which really isn't that surprising of a conclusion.
Corey Pronman is an author of Hockey Prospectus.
You can contact Corey by clicking here or click here to see Corey's other articles.