A generalized linear mixed model approach to estimating fumble frequencies in the National Football League

I told myself I was done with with Deflategate – and really, I was – that is, until I read this.

“Now I actually have some validation in the field,” Sharp said. “‘Hey, this guy was right all along.'”

Wait, what?

Forget the data twisting and statistical errors of the original analysis. The author claims to be vindicated by the fact that the Wells report found Patriots quarterback Tom Brady to be ‘more likely than not’ to have been involved with the deflation of footballs.

Okay then.*

*******

But despite my skepticism regarding Sharp’s analysis, two of the brightest minds in football analytics also taken the time to look at Patriots fumble rates, eventually concluding that the Patriots were indeed outliers.

First, after comparing Sharp’s critics to Nabisco running a study on snack cookies**, Brian Burke used multiple linear regression to model the number of fumbles in each NFL game since 2000, finding that the Pats posted much lower rates than the rest of the league in the years following 2007. Next, Benjamin Morris argued that the likelihood of team fumbling rates being at the Patriots levels or lower to be about 1 in 10,000. Linking low fumble rates and Deflategate findings, Morris writes that it “makes it more likely that the relationship between inflation levels and fumbling is real.”

One thing that Morris argues for – which I agree with – is that “there’s definitely more to be done on the Patriots fumbling to isolate for the fact that they were the most consistently winning team, the types of plays they ran.“

As Morris indicates, and what Burke hints at, is that modeling fumble rates is not straightforward, nor close to it. Because NFL teams aren’t randomized to run the same plays with the same time on the clock and from from the same spot on the field, any finding through this point has been evidence on the aggregate, averaged over games, plays, or perhaps a few in-game variables.

A play-by-play analysis, however, is missing.

And while it doesn’t ‘vindicate’ any particular finding, nor leave the Patriots free from suspicion, I found the task of looking at NFL play-by-play data to determine fumble rates quite interesting.

*******

I took the last 15 years of play-by-play data from Armchair Analysis (AA). All the code is linked here: the data costs $35, so I can’t provide that, unfortunately. However, if you have AA’s data, feel free to play around. Also, I’m going to focus on data from 2007 onwards. If you are interested in contrasting whether or notPatriots fumble rates changed substantially at any point over the last 15 years, I’d recommend a change-point analysis.

Let’s start with some descriptive statistics.

Point 1: Teams are less likely to fumble on QB kneel downs.

It’s easy one to begin with.

In fact, you are probably laughing right now, and you should be. There have been 5284 NFL kneel-downs since 2000, and not a single one resulted in a fumble using AA’s data. So who cares?

Here’s a plot of the teams who have taken the most kneel downs since 2007.

More than 25 snaps ahead the second place team, the Patriots have the most kneel downs.

Mentioning kneel downs seems silly, but this matters. Including kneel downs in an analysis of fumbles per play inflates the denominator (number of total plays) among teams more likely to be taking a knee, as the Patriots apparently were. In fact, the correlation between fumbles per play and kneel downs is -0.6. Here’s the relationship between the two variables. Teams with lower fumble rates tend to take more kneel downs (for one of several reasons).

After making the graph above, I deleted these plays. I also deleted QB spikes (Patriots had more of these than the average team) and any pass that was intercepted (Patriots had fewer than average). It’s hard for the offensive team to fumble on these plays. It’s even harder to fumble on kneel downs.

Point 2: Teams are less likely to fumble when they have the lead.

This was a bit surprising to me. For the regression model, I characterized each play based on the possession lead (3+, 2+, 1+, 0, 1-, 2-, or 3-) of the team with the ball. For example, an offensive team leading by more than 16 points would be up by three or more possessions.

Like kneel downs, scoring differential matters. Teams with the ball up by three possessions or more fumble more than 20% less often than other teams with the ball. So let’s see which teams have run the most offensive plays while up by three touchdowns.

Again, the Patriots show up, with nearly three times the median number of plays when holding a three possession lead. Again, this matters. To generally contrast New England’s fumble rates with Cleveland’s, when the Patriots have run more than 11x as many plays with a 3+ possession lead as the Browns, is silly. Teams fumble with the ball less when they are leading on the scoreboard.

Point 3: Yard line matters

Given the tighter window with which to run a successful play, it stands to hold that teams would fumble less on plays close to their opponents end zone. So, similar to points 1 and 2, any aggregated analysis of fumble rates could abnormally penalize teams that run a disproportionate number of plays in this area. Here’s the number of goal-to-go plays for each team since 2007.

The Patriots have run nearly 200 more goal-to-go plays than any other NFL team since 2007.

*******

Hopefully we can agree that not all plays are created equal. So how can we account for all of these factors?

Using hierarchical generalized linear mixed models (GLMM) of binary data via the lme4 package in R, I modeled the log-odds of a fumble occurring (Fumble = Yes/No) as a function of several play and game specific factors that are conceivably associated with fumble likelihood.

A hierarchical mixed model is advantageous for a few reasons. First, we can account for game conditions (such as the weather), play conditions (like down, distance, and yard line) and play characteristics (run left or pass deep right, for example) that may dictate fumble rates. Next, instead of model with several dozen fixed effects for each team’s offense and defense, we’ll use random intercepts for both the offensive and defensive units. Of particular interest will be the random intercept for New England; if this intercept is extremely low, it would provide evidence that after accounting for all the game and play specific variables, the Patriots fumble rate remains mysteriously lower than other teams. We can also test the significance of the random intercept for each team – if it is variance term is significantly different from 0, it would provide evidence that there remains substantial variation in the fumble rates driven by the team with the ball or the team on defense.

Please note that some of these results mirror a live-tweet version of the model that I ran in late January, but please check out the R code for how I decided to syphon things like down & distance, etc. These decisions were not easy, but were made with the intent of identifying what characteristics of each play might determine fumble outcomes. Here are the fixed effects included in the GLMM:

Score, Play direction, Final Minutes (Y/N), Playoffs (Y/N), Weather/Surface, Goal to Go (Y/N), Home team on Offense (Y/N), Goal to Go (Y/N), Down/Distance, No huddle (Y/N), Shotgun (Y/N), over/under, and spread.

And here are the random intercepts***:

Offensive Unit, Defensive Unit

And here’s the code. Model results are here:

fit.rush<-glmer(Fumble10~Score+playcall+FinalMins+Playoffs+Weather+GoaltoGo+OffHome+
 DownDistance+sg+nh+ou2+spread+(1|off)+(1|def),data=filter(pbp,type=="RUSH"),
 control=glmerControl(optCtrl=list(maxfun=300)),
 verbose=TRUE,family=binomial())
summary(fit.rush)

fit.pass<-glmer(Fumble10~Score+playcall+FinalMins+Playoffs+Weather+GoaltoGo+OffHome+
 DownDistance+sg+nh+ou2+spread+(1|off)+(1|def)
 ,data=filter(pbp,type=="PASS"),control=glmerControl(optCtrl=list(maxfun=300)),
 verbose=TRUE,family=binomial())
summary(fit.pass)

*******

The first thing we’ll look at is a plot of random effects for each of the GLMM fits. On the left is passing plays, on the right, running plays.

Once you account for play and game characteristics, it is really difficult to distinguish between the fumble rates of NFL teams.

In looking at passing plays, the random intercept terms for each offensive team are not significant predictors of fumble rates. The Patriots ranked as third in terms of teams least likely to fumble, given our model’s parameters. No teams intercept is noticeably different from 0.

There’s slightly more descriptive ability in using random intercepts with rushing plays. The Patriots’ intercept lies the furthest from 0, but it is not noticeably different from teams like Indy, Jacksonville, and Atlanta, which also boast significantly lower rates of fumbling on running plays.

Interestingly, Washington has the highest intercept on both rushes and passes.****

*******

If you are still reading, it is greatly appreciated. Mixed models have been used in awesome ways to answer really good questions in sports (see catcher framing and deserved run average for recent examples).***** This is not one such awesome application.

However, we learn in Introduction to Statistics that two variables are often associated for reasons beyond a causal mechanism. Given the results here, it seems safe to say that part of the link between the Patriots and low fumble rates was driven by game and play-specific conditions that those two variables were also associated with. Further, its easy to forget about funny data quirks in nearly all applied work, as we noticed with kneel downs and spikes in the football play by play data.

*******

Footnotes:

*There are a few other issues to consider. First, the Wells report also proposed that the Patriots started purposely deflating footballs in mid-2014. So, any lower fumble rates prior to this would have been, in relative terms, within league rules. Further, there’s also the issue of whether or not the Patriots ‘deflater’ travelled with the team, which unfortunately goes against the author’s inclusion of all games simultaneously. I can’t believe I just wrote the word ‘deflater.’

**This comparison seems ironic looking back, given that the NFL hired Exponent for its Wells report. Exponent was once was paid to argue that secondhand smoke did not cause cancer, among other suspicious claims.

***You may be asking yourself if we should be including effects (intercepts) for each running back. This is a fair question; if we include running backs as intercepts in the model, all team intercepts go to essentially 0. Given the RB’s are not randomized to carries, any team that purposely avoids playing running backs with high fumble rates would be penalized in our current fitting strategy.

****As a final step, I looked the significance of the random intercepts, given that from a model building standpoint, it’s generally preferred to use a model as parsimonious as possible. Including the random intercepts for both the offensive and defensive units significantly improves the model of fumbles on running plays, as judged by comparing the BIC of models with each random intercept to those without. On passing plays, the intercepts should be dropped from the model; there’s no evidence that, after accounting for game and play-specific covariates, teams’ fumbling rates differ from one another on passes.

*****A Bayesian strategy is also easy to implement. My guess is that a prior on team by team intercepts would only work to drag each team closer to 0.

*******

UPDATE: Scott from Football Outsiders requested that I include year effects as opposed to aggregating data across every season. I nested intercepts for each year within each team; this would account for seasonal trends for each unit, incorporated within some larger team effect. The yearly effects were indeed significant, both for offensive and defensive units. 

Here’s the plot of the random intercepts for each team, after accounting for seasonal trends.