It’s a fundamental question. It gets at the degree of stochasticity in the NHL.

The theme of this blog is that stochasticity is underestimated by hockey experts.

How much of what happens in cold ice rinks in the NHL is actually random vs. predictable? Well, if we lean toward the random, we undermine the GMs, scouts, and talking heads both at NHL Radio and among the throngs of analytics people. If we lean the other way, we make a case for a deeper dive into more and more arcane data.

There are few places in the hockey world that are higher stakes than the amateur draft. This is where the focus of debate often comes home to roost. A team that can get an edge on drafting good players has a distinct and significant advantage.

Wasting a valuable high draft pick on a player that never pans out is a franchise disaster. Clubs that pick successful players in the fourth rounds and beyond are ascendent.

So, let’s take a big picture view. I present a data picture below that plots the overall draft position through all seven rounds in the NHL draft in the modern era (after the lockout) against the number of points that player has scored per twenty NHL games played. Players that have not scored or played in any games get a zero for reference.

DATA SOURCE: hockey-reference.com (accessed 12/5/17)

Now, there are at least four interesting things to say about this graph, and they are all deep and important.

The scouts are pretty good at picking talent among those drafted in the first fifty positions. This is an exponential distribution. It shows that scouts recognize the best of the best. The top ten players picked score between 8–10 points per twenty games. This is clearly a sign that the best talent is recognized by the professionals in scouting.

The ability of talent detection and drafting processes drops off dramatically at about position sixty. It’s striking that the points per twenty games is quite flat after that point. The scouts and teams are just not good at all at sorting anything but elite talent.

This has many implications. Most importantly, it suggests that drafting after about sixty is largely a crapshoot. If anyone could figure out how to differentiate these players, they would be worth a great deal. There is enough dramatic variation in the data to suggest that whatever pro scouts are doing, it does very little to explain what is going on after about position seventy.

Perhaps this is not surprising. The pro scouts will spend most of their time talking about the same set of elite prospects, ignoring details that differentiate players across the rest of the distribution. However, overall team success depends on picking players in later waves of the draft. Note that the degree of variation in performance is nearly random after about position seventy. There are players drafted after position 150 who score as many points as those that are drafted in the top 20.

This is very noisy data. Sure, lots agree on the top 40. But the spread of player performances across the range of draft position suggests that largely, talent scouting in hockey is a very imprecise business. This is more of a random cloud than a well-honed talent scouting methodology.

Notice the very dark and thick line on the 0 point on the vertical axis. The players who line up on this edge are those who are drafted into the NHL, and never score a point or never play a game in the best league in the world. What is important about this is that there are about as many players who never score a point in the NHL who are drafted in the top 50 as those who are drafted in the sixth and seventh round. Elite players tend to play well but there are plenty of players identified as elite who never pan out as well as players drafted after round three who end up being elite.

We need a better predictive model to reduce this noise.