Calculating the value of a draft pick is not an easy task – there’s a great amount of uncertainty involved in any given pick, and the number of “misses” are significantly more than the number of “hits”, particularly in later rounds. Compounding the issue is the lack of a clear definition for success – while most methods to date have defined a successful pick as a player who reached the 200 NHL game mark, this approach is flawed in that it ignores differences in contribution and is biased against higher round picks. In this article, we’ll look at how to address these issues with a few simple changes to the standard approach, and use a novel evaluation approach to build a model that should more accurately estimate the value of an NHL draft pick.
Not All Success Is The Same
The primary limitation with traditional approaches to draft pick valuation is that they tend to view all players who become NHL regulars as equivalent. Regardless of whether a pick turns out to be Sidney Crosby or Alexandre Daigle, most past studies have viewed them as equals since they both reached 200 NHL games. This, of course, doesn’t reflect the reality of the situation – Crosby plays more than 20 minutes a night for the Penguins, while Daigle rarely reached more than 10. This lack of differentiation between levels of success is problematic, particularly when finding the relative value of picks within the first round, where the drop in talent between the start and end is often steep.
Bias Towards Players Drafted Earlier
The other drawback of existing methodologies is that they fail to account for the fact that earlier picks tend to get rushed to the NHL faster, and have an easier time reaching the 200 game mark as a result. Since players chosen in the earlier rounds are often viewed (for better or for worse) as the future stars of their organizations, coaches and general managers are more willing to give them ice time in the NHL, regardless of their performance. On the other hand, a late round pick may be given relatively fewer chances to make the big league club before they’re marked as a career minor leaguer. Even if both players end up in the AHL five years after they’ve been drafted, a first round pick will have a much greater chance of getting to 200 games than a fifth rounder.
Changing the Variables of Interest
Knowing that the 200-game measure is flawed, how can we choose a metric that better accounts for the issues noted above? The first thing we can do is to change the time period over which we evaluate prospects. Rather than considering entire careers post-draft, we can use a window further down each draft pick’s career to ensure that all players have had a chance to develop, which should help to eliminate any potential bias against later round picks. The average player has their rookie season at age 22, therefore if we use a draft pick’s performance between age 23 and 25 as our sample we should be able to evaluate each player on the merits of their talent, rather than their perceived value out of the draft.
The second change we can make is to look at average ice time, rather than total games played, to determine the success of a draft pick. Average ice time allows us to calculate an estimate of what line/pairing each player would be placed on, based on historical usage for NHL teams, and provides a more accurate view of their value than games played. If we calculate the time on ice required to be a 1st/2nd/3rd/4th line player on each NHL team historically, and then set a cutoff at the 90th percentile for each line, we get the following estimates for the minimum minutes played by a player on a given line:
|Line/Pairing||Minimum Minutes – F||Minimum Minutes – D|
Estimating Draft-Pick Success by Draft Position
With our new metric and sampling method defined, we can now look at predicting the probability of success for a player selected at each draft position. Our sample will be all players taken with picks 1-210 in the entry drafts from 1993 to 2007. Our cutoff years are defined by the data available on NHL.com – we can’t include those taken before 1993 since the NHL’s data, including average time-on-ice, doesn’t go back further than 1997. Further, we can’t include players drafted after 2007 since they have yet to play their age 25 season. To ensure that we’re capturing regular NHLers, we’ll only include players who played an average of 40 games per year in at least 2 years of their 3-year sample. Lastly, we’ll exclude goalies, as the number of goalies drafted is relatively small, and the nature of the position doesn’t lend itself to evaluation using average ice-time.
Our inclusion criteria leaves us with 3387 drafted skaters, comprised of 2027 forwards and 1160 defencemen. Of those included, 375 forwards went on to play at least 4th line minutes between age 23 and 25, while 171 defenceman reached the 3rd pairing level over that same period.
Our evaluation will look at forwards and defencemen separately, in order to capture differences in value by position. We’ll use multinomial regression to predict the probability that a player drafted at a given overall draft position will become a 1st, 2nd, 3rd, or 4th line forward, or a non-regular. We’ll use the same method to predict the probability that a defenceman will become a 1st, 2nd or 3rd pairing defencemen, or a non-regular. Our independent variable will be 1/sqrt(n) where n is the overall draft position.
Looking at our results in the graph below, we see that for forwards, the first overall pick was a near-sure thing from 1993-2007, with the predicted probability of selecting a 1st line player reaching nearly 90%, and less than a 1% chance of finding a non-regular. As we would expect, the probability of drafting a first line player decreases rapidly as the first round goes on, and by the 15th pick teams are more likely to be selecting a 2nd line player than a 1st. After the 2nd round, the draft becomes more difficult to predict: the odds of finding an NHLer of any level drop below 15%, and the distribution of likelihoods between lines becomes pretty much equal – you’re nearly as likely to end up with a 4th line player as a 1st line player after pick 60.
While forwards have a degree of certainty in the early picks, defencemen are a bit more of a gamble, with the 1st overall pick expected to turn into 1st-pairing blueliner roughly 70% of the time. Similar to forwards at roughly the half-way point of the first round, your odds of drafting a 1st pairing player and a 2nd pairing player converge. Unlike forwards however, defencemen become a lot more uncertain a lot earlier, with the odds of drafting a regular NHLer falling below 10% around the end of the 3rd round.
Now that we have estimated the likelihood of getting a player of a given talent level with each pick, we need to figure out what a player in each of our line groupings is worth on the open market. If we look at all players 27 years of age or older who played at least 500 minutes in 2014-2015, and group them by their average ice-time as we did above, we can use their salary to estimate the open-market cost of a player on each line. Utilizing data from War On Ice, we find the average salary in each bucket to get the following cost estimates by line/pairing:
|Line/Pairing||Forward Cost||Defenceman Cost|
Next we can multiply the cost of a skater playing on a given line by the probability that a pick will end up playing on a given line to calculate the per-year value of a given pick.
In general, the value of picking a forward at any given position is higher than picking a defenceman, because forwards are typically paid more, and generally more likely to turn into regular NHLers. The Weighted Average line represents the average value based on a 2/3-1/3 split in picks between forwards and defencemen (this is roughly in line with the split in our sample). Although the gap narrows to less than $100,000 once you get past the late third round, the difference in expected value for the first overall pick is rather large, with a forward being worth over $1MM more than a defenceman taken in the same spot.
We should note that after applying the differences in player cost by ice-time, the gap between the first overall pick and all others becomes more extreme. If we had simply applied an average salary to each player’s overall probability of becoming an NHLer (i.e. if we had ignored the differences in playing time by pick, and value by line), we would have undervalued the first overall pick by between $0.7MM and $1.7MM while overvaluing every pick outside of the top 25.
The Discounting Problem
One caveat to note with this approach is that it predicts the value of a pick in future dollars, rather than present-day dollars. While this generally isn’t an issue when making decisions on relative value (i.e. is the 3rd pick worth more than the 10th and 20th pick combined), it does make a difference when the decision is between current players and future players (i.e. trading Sidney Crosby for the first overall pick). This draws attention to the broader issue of how much value a future win has versus a win today, and should be the focus of a much more detailed inquiry. Given that few players drafted outside of the top 3 receive significant ice time in the year or two following the draft, it’s reasonably safe to assume that few teams are trading for draft picks in order to win in the near-term. So while the problem of discounting future wins is certainly interesting and worthy of further study, our estimates here should suffice for most practical purposes.
For those of you who are interested into digging into the numbers a bit more, I’ve uploaded the estimated value by pick number for both forwards and defencemen here.