In a previous post, I argued analytics ought to be viewed as a method of solving problems and that the manner in which it has been mainstreamed by recent case studies (notably the Wild, Kings and Leafs), is problematic. That is, analytics has been mainstreamed under the framework of "analytics as possession," which has, in part, encouraged a variety of unproductive arguments, esp. those assuming analytics fails to account for a variety of outcomes and/or variables.
In this post, I'd like to examine a counter case study, which I hope will highlight the way analytics at its core is about solving problems and not about any specific tool, number or combination of tools and numbers.
The New Jersey Devils are an outstanding puck possession team. Here are charts of the last two years showing where the Devils stack up 5x5 by FenwickClose (3rd, 5th):
(both charts via extraskater.com)
The Devils are in elite company among possession teams. And, the vast majority of these teams are winners. Or, possession correlates very high with winning hockey games.
When we look at where the Devils sit in the standings, however, it is apparent that their elite-level possession is not translating into an elite-level win record. Here's their current spot in the standings:
In an environment where the Leafs possession failures are dominating the mainstream analytic conversation, the Devils present an interesting case study. That is, the way in which the Leafs stats/anti-stats debate has been framed is misleading. It has constructed a hyperbolic situation in which "analytics" is synonymous with possession and whereby the validity of analytics is dependent upon the success/failure of possession to accurately predict outcomes.
To the analytics community this hyperbolic situation is surely absurd. To the nay-sayer and the casual observer, however, no doubt this is how they have encountered analytics: as a question of possession and prediction.
To this group, I think, perhaps the best way to demonstrate analytics as a method, rather than a set of tools or numbers, is to show how it might approach a different kind of problem. The Devils represent a different kind of problem insofar as they invert the situation of the Leafs (they are an elite possession team faltering in the standings).
To a naif, surely the mere fact of the Devils ought to debunk analytics. If possession can't account for the Devils' losing season, how can it account for anything?
And yet, analytics as method merely sees this as another problem to be solved.
So, what is wrong with the Devils?
To start with, they have been extraordinarily unlucky. They enjoy a league high 16 overtime loses and are an incredible 0-11 in shootouts.
They are 29th in PDO.
But, it's not all luck.
Another part of the problem is roster construction. While built to dominate possession, the Devils are a very poor shooting team. Now, Edmonton Oilers fans will recall fondly the debate about Jordan Eberle's unsustainable shooting percentage. However, over time the shooting talent of a given player settles into a sustainable historic average. In the case of the Devils, especially without Ilya Kovalchuk, we appear to be witnessing a team constructed without a lot of true talent shooting ability.
Beyond luck-like categories like shooting and save percentage, however, there is another problem with the Devils.
While they are 1st in the league in fenwick against per 60 minutes, they are 27th in fenwick for per 60 minutes. That is, while they are exceptional at limiting unblocked shot attempts, they are very poor at generating their own.
Sure, they generate enough to cover the tiny amount they allow such that their fenwick percentage is phenomenal. However, this percentage fails to acknowledge that when the Devils are playing neither team is generating many shots. This is simply a product of the anti-entertainment brand of hockey the Devils employ.
If we compare these shot splits with other elite possession teams, say the Chicago Blackhawks (2nd in 5x5 FenwickClose%) and the San Jose Sharks (3rd in 5x5 FenwickClose%), we see a very different picture:
San Jose is 6th in shots against per 60 minutes (7th in FA/60) and 3rd in shots for (1st in FF/60).
That is, not only are these teams winning the percentage of unblocked shot attempts battle (fenwick), they are also playing a style that generates a lot of shots on net. The Devils, however, simply aren't constructed and/or deployed to generate enough attempts to score goals.
This is hardly an exhaustive look at the Devils (hint: go here for more in-depth coverage). We haven't even covered what's weird about how lucky this team has been on special teams
What's weird about the #NHLDevils? 1st in 4x5 SV% 4th in 5x4 SH% great 5x5 team is unlucky 5x5, but very lucky on special teams. nuts.— Romulus' Apotheosis (@RomulusNotNuma) April 2, 2014
What I'd like to suggest, however, is twofold.
First, analytics is best understood and deployed as a kind of method of problem solving and not as a set of tools or numbers in the abstract.
Second, the way we talk about analytics and the case studies that get mainstreamed affect how analytics will be perceived and understood. In order to complicate the framework in which the current mainstreaming of analytics has taken on, alternative case studies should be sought that can showcase some of the other problems analytics can solve (at least provisionally).