The NFL draft – where we stand in 2016

Another NFL draft has come and gone, and with it has come the predictable displays of unyielding optimism, stale and arguably race-based generalizations of player skill, and, as a relative newcomer in 2016, lazy misuse of the term analytics.

In following along this spring, it became clear that what is mainstream knowledge among researchers is far from it in the national media. This despite a decent amount of both academic and non-academic research into the topic.

For those new to the scene, or even for a few veterans who may have missed an article or two along the way, I decided to write a quick review of what’s out there. Note that many of the following points are related to one another.

1. Top draft picks are overvalued. 

In an efficient market, the value of picks traded between teams would be equivalent.  That is, if Pick X was traded for Picks Y and Z, the value added by picking in spot X should equal that of picking in spots Y and Z. Interestingly, in a pair of well-regarded papers from researchers Cade Massey and Richard Thaler, the authors identified marked inefficiencies in how NFL franchises value different draft picks.

Turns out, the preferable solution is almost always to acquire more picks. In our example, better to have Y and Z than X alone. It’s why the the common suggestion among many who have studied the draft is that because teams overvalue earlier picks, it is generally a good idea to trade down. The edge goes to those who take advantage of the overconfidence shown by others.

Related: go Browns.

2. Top picks tend to be (relatively) overpaid.

In addition to being overvalued on behalf of league officials, players picked at the top of the draft have also tended to offer less of a return on their investment than those picked at the end of round one. To be specific, the NFL rookie pay scale is nearly fixed, and players picked near the top of the draft get paid substantially more. As a result, for several years, it was optimal to pick late in the first or early in the second rounds, where the expectation was to obtain a decent player with a substantially more palatable contract. See Figure III in Massey & Thaler’s paper here, as one example. The author’s aptly call this a loser’s curse; bad NFL teams are stuck picking at the top of the first round, where the expected return is actually lower.

Note that given the new CBA which updated rookie contracts in 2014, it is less clear if such a surplus value still exists in today’s NFL.

3. Teams are, by and large, making guesses

One of the most consistent findings in current literature is that a team’s prowess in picking good players doesn’t translate from one season to the next. Chase reached such a conclusion a few years ago, and I really liked Neil’s scatter plot below, which shows the lack of a year-to-year consistency in team-level returns over expectation.

 

To sum, despite all that you’ll hear to the contrary, there’s no evidence to date that an organizational-level ability to identify talent in the draft is repeatable.

And if team’s are unable to consistently identify good picks in the draft, it’s safe to say that draft pundits can’t, either. In fact, when I compared Mel Kiper’s draft grades to his re-grade of that same draft after a few seasons of play, the link was wink. That is, Kiper’s post-hoc analysis of a draft barely drew any of the same conclusions as his first one.

So, whenever you hear team officials boasting about an incoming player – or even when looking at the so-called draft ‘grades’ put out everywhere – it’s not much different than listening to someone saying they like heads in a coin flip or red on the roulette table.

4. Sunk costs make analysis difficult.

Let’s take an interlude to talk fantasy football. Imagine that you drafted Peyton Manning early in 2016, making a substantial investment in a quarterback who you anticipated would lead your squad to greener pastures. Turns out, Manning stunk from week 1 onwards. But there he was in your week 5 line-up, then again in your week 10 line-up, and so on. You couldn’t move on!

Turns out, NFL organizations also suffer from the same problem. Such was the conclusion of Quinn Keifer, who found that players chosen at the end of Rd. 1 received substantially more playing time than those who were picked at the beginning of Rd. 2. In other words, the premium made on first round picks was responsible for eventual increases in games started. Keifer blames sunk costs, in which prior investments have led to teams making irrational decisions down the road. This is also what ruined any fantasy player who stuck with Manning.

This result matters when it comes to analyzing player success. If organizations are afraid to cut loose players that they drafted early, it would muddle any link between draft round and player success, and it could artificially inflate the link between early draft picks and long term success.

5. What else is out there?

I explored a few major findings above, but here is a brief synopsis of other interesting material.

-There may be position-level variability as far as identifying talent. Of course, this becomes complicated at some positions, such as linebacker. Write Zach & Rob, “It’s also possible that players with good pedigrees and name recognition are being given preferential treatment when awards are granted, in the absence of meaningful performance data (like what exists for offensive skill players) to challenge our perceptions.

-Neil and Alison look at position-level value here.

-Sports Analytics groups out of Carnegie Mellon and Harvard have done some solid work.

-There are oodles of NFL draft value charts. Examples include the original one credited to Jimmy Johnson, newer ones from Chase Stuart and the Harvard cohort, and perhaps the most methodologically driven edition from Michael Schuckers.

-In a more recent article, a pair of authors conclude that picks made from teams that trade up tend to outperform picks made when teams stand pat, as judged by games started and approximate value in a player’s first three years. However, my initial reaction to this work is one of skepticism, in large part because of what we went over earlier about sunk costs. If teams trade up for a player, that investment could lend itself towards a team aggressively showcasing him. Additionally, the author’s do not mention the often prohibitive cost of trading up, which could offset any gains in surplus drafting.

-In the next few days, I’ll have some posts that look at the history of the draft and team success. More to come.

So what did we learn?

It is extremely difficult to consistently identify talent in the NFL draft. However, there are inefficiencies, which generally arise when teams overvalue players by trading up to acquire them.  Such displays of confidence belie a process which is, by and large, no different than picking heads or tails before a coin toss.

Perhaps Massey said it best in an article on Vox:

“It’s basically a coin flip, but teams are paying a great deal for the right to call which side of the coin.”

 

Advertisements

4 Comments

  1. Excellent article. I generally agree with trading down, but I want to look at the instances where there is an argument for trading up.

    I will ignore the financial aspect of “marketability” and “jersey sales”. While that may be a factor for owners and GM’s, we’re examining how it influences team value and wins.

    I have done some work looking at 3+ all pro players and it’s the only metric that suggests trading up to the top 5 picks as opposed to trading down va the Jimmy Johnson chart.
    Any other metric such as 1+ pro bowlers, % of 1 yr starters by position, % of 3+TE starters, % making 2nd contract with team, etc all point towards trading down. So there has been in the past a dramatic drop off in % of players that get voted 3 or more times to the all pro. These are usually future hall of fame type of players, the best at their position for years of dominance.

    If your goal is maximize Super Bowl chances at some stage of the rebuilding cycle, rather than total wins… There are a few instances where you may actually prefer a high stakes gamble with a small chance of a huge payoff than a large chance of a small payoff, even if the 2nd one adds more “value”.
    Imagine a 2 player team composed of numbers. Super Bowl teams are typicall rated 180 or higher.. A 99 overall player with a 10 overall player is worse in year 1 than two 60 overall players, but perhaps better overall in terms of building your chances at a Super Bowl because it will be easier to find an 81 or better player in year 2, than 2 90+ players. Add to it that a 91+10 would get to draft higher in year 2, and selling out for a small chance at an all star may make sense.

    The evidence of trading up sometimes possibly being better, is very nuanced and complex, but it would be the same concept of “utility theory” demonstrated in the example where you only have 2 players.

    Although the Harvard Sports Analysis Chart (HSAC) suggests having a handful of 7th rounders is more valuable than a single 1st, it would also (using the same logic) rank a choice of a dozen undrafted players extremely high as well. Except you have very little cost to acquire an undrafted player.

    In essance, having a roster filled with a say 2 dozen undrafted players would be better than a dozen 7th rounders which would be better than a single 1st rounder. But those aren’t the real choices you’re making.

    Realistically, that’s not an option. You have to have a 90man roster, and so each additional acquisition requires you to either eliminate a player on your roster, or have one less undrafted player.

    So, theoretically a team filled with potential all pro starters everywhere should either trade for future picks, or possibly consider trading up for the simple reason, they have fewer spots to fill on the 90man, and greater costs to replace a player on the 53man that requires a home run to even have a chance to add value.

    To an above average team, this still applies, to a lesser extent. Perhaps trading down is less bennefial, or not beneficial, and trading up is less bad, or possibly good in later rounds.

    Certainly there is at least a marginal advantage for drafting higher. The question is not “Is moving up in the 4th worth giving up a 5th?”. Instead the question is something like, “Does an undrafted player plus pick 103 overall, give me a better chance than pick 119 and 151 in achieving the goals set according to the timeline set”? If so, trade up.

    Since teams are not all created equal and different needs exist, there are some scenarios, where even using the same metrics and drafting from the same positions, they may answer that question differently, even if the goal was “maximize wins this year”. To a team with a single weak point, they may want to trade way up, and then fill a few later rounds or undrafted players at the same position as that weak point. Afterall, they probably have only room for one at that position to be on the field at a time.

    Also, there’s a question of how to evaluate players you have ranked as a 1st rounder that are still available in the 4th. The draft is probably filled with examples of the majority of teams believing they got value at the draft position, and also those that couldn’t trade down without accepting drastically less. Should you then trade up? How extreme does your evaluation have to be? If it’s practically a coinflip on whether or not you outperform, is it still possible that a player fits schematically in one system, but not another? Perhaps Improvement of skills by opponents and replacement of poorly skilled in organization combined with changes in schemes and systems due to highly volatile, small edges players have in one scheme, but not another, can somewhat explain drafting talent? Perhaps a RB has tremendous value simply because very few teams use a particular system, and the RB has specific skills and experience that only fit one system? (the Kubaic/Shanahan system RB of ‘”one cut and go” comes to mind).

    Either way, statistics must be used to objectively evaluate any hypothesis, and drafting based upon some untested idea without close examination is not advised.

  2. Covariance is always the bugaboo of these types of analyses. One of the biggest issues is that teams with high draft picks are often trying to find a ‘star’ player such as a quarterback. This places an incredible amount of pressure of putting a rookie into a highly visible, high pressure situation. NFL wisdom used to be that a quarterback takes 2 to 4 years to develop into a starter. One of the big pressures on GMs is getting the star player to fill the void. Once you make that investment, the media and the fans want to see the star play. But for every Matt Ryan there are 10 RG IIIs. In the case of QBs especially, they are going to bad teams. So you’ve got a rookie trying to learn the offense, play at the speed of the NFL, with tons of pressure on him. This is going to exacerbate the likelihood of failure considerably. There are probably 20 good NFL quarterbacks who could take a good team into the playoffs and 32 teams. Few coaches have the luxury of understanding fans and owners with the patience necessary to develop a winning long term strategy. And all of this is complicated by injuries, free agency, and who else is in your division and how well they are doing.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s