by Robert Vollman
The primary design flaw of GVT was felt to be that some of the values in the calculation were “arbitrary”, though that word was perhaps a shorthand way of saying that the designer (Tom Awad) chose them based not on firm, objective calculations, but rather ones that included a significant measure of reasoned opinion. In that regard, GVT is clearly unlike most other statistics, and was a focal point of the discussion.
To that point, the main defense for the continued use of GVT was that no statistic is without flaw, and that having those flaws doesn’t make them useless. GVT was even compared with baseball’s WAR, which is seen to have similar flaws, but nevertheless serves its purpose sufficiently well to achieve relatively widespread use.
To these points, the main counter-argument is that while statistics don’t need to be perfect, they need to be much closer to perfection than GVT. Why? The main concern was that GVT’s flaws were more serious than those of other statistics, and/or were harder to explain to others, and therefore more prone to misuse and wrong conclusions. Furthermore, the concern was that analysts using GVT would fail to mention its proper context and shortcomings and use it as a crutch, and as “discussion enders” instead of “discussion starters”. There was also a concern that all of this would keep some people away from the field of statistical analysis.
Since this part of the discussion was primarily opinion-based, the consequences were largely in the hands of how the analyst actually uses GVT, and ideas that potentially required from than 140 characters to sufficiently explain, there wasn’t much to add.
There was one other side-discussion that also briefly took place. The other criticism of GVT was that its use of goal-based data (goals, assists, adjusted plus/minus) renders it less accurate than if it had used shot-based data. However, this criticism was likely not targeted at this particular application of GVT (the Mark Streit tweet), which used five years of data, at which point the difference between goal-based and shot-based data would be minor.
But more than that, taking advantage of some more modern advances, like penalties, shooting percentage, player-specific team-on-ice metrics (and so on) could produce a superior version of GVT (although one that could only be applied to modern-day NHL, never to other leagues, nor to 20th century NHL seasons). This being true, how come such a statistic hasn’t already been designed? There was a feeling that a superior alternative to GVT could be easily designed, but since it would likely still fall short of the “near-perfection” bar, that such design efforts were best focused elsewhere.
Promisingly, there was an obvious consensus that a basket of player usage and shot-based statistics could much more accurately measure a player’s value than any single high-level, all-in-one statistic like GVT, but the primary disagreement was whether or not that meant that GVT should be used at all. Of course, the point could probably be conceded that it would be difficult to use such a basket of statistics in a 140-character tweet approximating the value of Mark Streit’s contract.