Stat pundit rankings: 2014 NFL win over/unders

We are back for another edition of the stat pundit rankings, where we rank the accuracy of different predictions for team wins from statistics or simulation based websites. Team Rankings boasted the best performance last year, outperforming competitors and the totals set by sportsbooks as far as predicting 2013 regular season win totals.

Let’s meet our competitors for 2014:

Team Rankings (TR), predictions listed here

Accuscore (AS), predictions emailed by a loyal reader

FiveThirtyEight (538), predictions extracted the week before the regular season began (missing link)

Prediction Machine (PM), predictions listed here, released just after the season began

Football Outsiders (FO), projections listed here from just before the season began

Aggregate, the average statheads predictions from the five sites above

Finally, we will want to compare all the projections to lines set by sportsbooks. To do so, I used the implied lines used by Seth Burn in his preseason post, done a week before the regular season, which accounts for both the sportsbook total and the vig. I label this in my graphs as ‘Vegas’ for simplicity.

I’ll also include last year’s win totals (2013) and a method that assigns eight wins to each team (Eights), just to see how things line up.

     1.  Which site boasted the most accurate predictions? 

To answer this question, we use two metrics, the mean absolute error (the average distance between a team’s observed and expected win totals) and the mean squared error (the average squared distance). The mean absolute error is more easily interpretable, but using mean squared error punishes places a harsher punishment on predictions which are further from the observed total.

By both metrics, Team Rankings is the only prediction site to outperform Vegas, doing so by about an eighth of a win, on average. Overall, sportsbooks were about 0.15 wins closer to the observed win totals in 2014 than in 2013, missing by an average of about two wins per team. This suggests that the 2014 season was slightly easier to predict than the 2013 one.

Here’s a barplot of both metrics for each of our predictors.

First, mean absolute error. All predictors except for the Outsiders were better at predicting 2014 that using each team’s 2013 wins.

Overall, using eight wins for each team leaves you with an average error of about 2.5 wins per team; the best statistics site that we looked at reduced this to about 1.90. The average of the statheads sites also did fairly well, with an average error right around the sportsbooks’ one.

Next is mean squared error. Everyone did better than just using last year’s win totals.

    2.  How would using the sites to place over and under bets do? 

Let’s look at how each site were do to have done if we were to have used their predictions to place mythical wagers on each team’s totals. Overall Team Rankings and Accuscore led the way, accurately predicting 20 and 19 of the 32 sides, respectively. FiveThirtyEight hit on exactly half of the 32 sides correctly, while Prediction Machine and the Outsiders only hit on 14 and 12 sides, respectively.

Looking at each site’s favorite predictions – that is, the five for which each site deviated from the sportsbook predictions by the largest absolute value – Team Rankings (3-2) and Accuscore (4-1) were the only ones to finish above 0.500.

The sites favorite successful predictions were the unders for Chicago and the New York Jets. On average, however, the sites also liked the under for New England and Green Bay, doing so unsuccessfully.

Most of the predictors accurately hit on the Minnesota, Houston, and San Diego overs, while missing on over bets for the Jaguars and Rams.

    3.  What’s the graph look like? 

I sorted each team by its predicted win total, and plotted observed wins and predicted wins. For predicted wins, I both used the sportsbook totals I described earlier, as well as the predicted wins for each team averaged across each of the five websites (called the ‘statheads’ prediction).

The graph makes it obvious the depths by which the Bucs, Titans, and Bears underachieved, while Cowboys, Cardinals, and Lions were among the biggest surprises.

4.  Is there anything else I should know?

Sure.

Several sites make weekly ATS picks, so let’s summarize the 2014 season. Here, I’ll focus on the websites that make it easily accessible to track their efforts.

Seth Burn tracked Football Outsiders weekly ATS picks, which finished an unofficial 111-136-9. This would put the Outsiders in the bottom 6th percentile of picks if we were to simulate all NFL game picks by flipping a coin. Seth also tracked the Outsiders selection using the Kelly criterion, where more money is mythically wagered on the strongest selections. Things did not go well, with the picks finishing well in the red. The results for 2014 follow a disappointing 2013 effort, in which the Outsiders finished 11 games below 0.500. The website finished above 0.500 during its first four years of making picks, from 2008 through 2012, with yearly finishes as high as 57% on all picks.

Massey-Peabody finished 36.5-30-1.5 on its official plays, which, while slightly down when compared to its 2013 performance, was still a reasonable effort.

Finally, Team Rankings finished at 31-24 on its most recommended plays of 2014 (56%), after finishing at just 43% in 2013. On all picks, the site finished 2 games below 0.500.

(Thanks to Greg over at statsinthewild.com for his ideas and help with this!)