538 is six months old…where does it stand?

Wednesday marks the six-month anniversary of Nate Silver’s FiveThirtyEight launch with ESPN.

As the site mixes statistics, sports, data visualization, and academic research, it’s been a must-read on nearly a daily basis for me.

Here’s my unsolicited and slightly ambiguous view of where things stand:

It’s really, really hard to do what FiveThirtyEight is trying to do and to do it well.

Here’s why:

FiveThirtyEight’s business model is primarily based on advertising, and advertisers generally don’t flock to sites that only spit out content once a week. As a result, Silver and colleagues are forced to put out articles at a frenetic pace. As one example, I estimated that FiveThirtyEight wrote 2.5 articles per day covering the 2014 World Cup, posts which were generally written by only one or two full time writers. That’s an incredible pace.

But here’s the catch; while the judging of a data journalism website should be based on content alone, and not the frequency of posts, the precision that is required and expected for this level of research can often take an inordinate amount of time. For instance, on his old website, current 538 sports data journalist Benjamin Morris wrote an outstanding set of articles on Dennis Rodman. You should read or skim these if you haven’t already, as its some really fascinating and convincing stuff. It’s no surprise that Morris could put together this type of series, given both his background and, importantly, the time and effort he was able to devote to the series. Based on postdates, the whole series appears to have taken at least 9 months for Morris to put together.

The problem now is that 538 can’t afford to wait a full year for a series of Rodman posts or something similar – data journalism has to meet deadlines, even when the data analysis can’t.

Let’s now move to Morris’ recent piece on Week 2 of the NFL season, linked here. Midway through, he describes some research that looks into the eventual success of rookie QB’s. He writes:

There are a lot of ways to run regressions on rookie [QB] stats to see how likely these players are to have good careers, but here’s a basic version with quick and dirty t-values4 for each stat:

To see which stats are predictive, we’re looking to see which have t-values higher than 2 or lower than -2.....

So what does that mean for this year’s crop of rookies? Every week that Teddy Bridgewater and Johnny Manziel sit is more bad news for their career prospects. From a purely predictive perspective, it’s better for them to play and have a bad game than to sit.

It’s better news for Oakland quarterback Derek Carr, who started in Sunday’s performance against the Jets (good), had two touchdowns (good) and no interceptions (doesn’t matter) in a losing effort (doesn’t matter), but only threw for 151 yards (bad). With Carr continuing to start while others continue to sit, his stock continues to rise.

There’s nary a statistician in the country who would let this through an editorial process.

There are many reasons that rookie quarterbacks who start more games turn out to be better quarterbacks. Most obviously, if you start more games as a rookie QB, you are probably a better QB to begin with! Results like this are an obvious case of selection bias, in which QBs that receive one treatment (starting role) differ from those that receive another (bench role) with respect to other traits (talent) that are also associated with the outcome (career success).

To suggest that games started helps to predict future success ignores the talent that the starting rookie QB’s have more talent to begin with.

Let’s imagine, for example, that I had written the following:

There’s a significant association between being drafted early and how well a QB performs.

Well, yes, that statement is obviously true, and it’s an association with the same issue as the “games started” comparison. But what if followed it up with this:

For predictive purposes, that means if Tim Tebow had been drafted higher, he would play better.

You’d call me insane if I wrote that.

Morris isn’t insane, and far from it. But it is misleading at best to argue that Manziel, Bridgewater, and Carr’s career prospects are dependent upon games started because over the last several decades, NFL coaches have usually been smart enough to play good rookie QB’s but not bad rookie QB’s.

Morris’ NFL Week 2 article was posted on 538 on Friday at 1:00 pm. I won’t pretend to know the ins and outs of a webpage’s business model, but I’m smart enough to know that there’s a big difference in pageviews between Friday at 1:00 pm and, say, Friday at 7 pm or Saturday at 7 am, particularly for a post on relating to the NFL’s upcoming weekend.

Additional time to write, an exhaustive editorial process, or the collaboration of a few writers on a piece like this may have yielded additional care in the word choice for the Rookie QB’s section.

But taking care, of course, might not make a deadline.

——————————————————————————

Other related comments that didn’t fit above:

1- In almost all cases, the graphics on 538 are well designed. It’s clear that lots of effort goes into these, and it is not unnoticed.

2- The site has really picked up its work on areas outside of sports over the last few months. Ask Mona keeps me on my toes, the eventual articles on Ferguson were well done, and I’ve enjoyed several of the pieces in the science and economics sections.

3- I’d love to know how much money was wasted in 538’s burrito contest.

4- I encourage readers to check out Alberto Cairo’s related comments here.

5- From a reproducibility aspect, I respect the fact that 538 links to and shares some of its data. Great for aspiring and interested researchers.

6- If nothing else, 538’s doing a much better job with statistics than Grantland (Ex 1, Ex 2)

7- I’d love to see 538 put some long-form pieces together (like Morris did with the Rodman series). It’s writers are creative and talented enough to change the way fans, coaches, and players think about sports, and longer and more in depth pieces could do that.

8- Final idea: The “Sports Analytics Blog” runs a great weekly round-up, which summarizes important sports analytics articles written during the previous week, using content from both major websites and smaller blogs. This is a great resource for weeks in which I don’t have the time to access content daily.

538 should copy/poach/adopt this model, particularly with an eye towards academic articles. For example, the recent article of JQAS just came out, and it’d be easy to link and summarize the key articles (especially because most people can’t get around the paywall). It’d be an easy post to write, if nothing else.

Postscript 1: Morris responds: “Fair enough, but I don’t think the rookie thing means to imply all that you suggest it’s failing to prove….i.e., my analysis of rookie QBs has always been acausal. E.g. http://fivethirtyeight.com/features/2014-nfl-preview-the-broncos-versus-the-hopefuls/ … or http://fivethirtyeight.com/features/worry-about-johnny-manziels-weight-not-his-height/ …

Postscript 2: Not sure how I forgot this, but someone needs to remind the boss the difference between probabilities and odds.

Postscript 3: Eric writes the following:

An interesting older study that might have something to say about your Tim Tebow comment:
Staw, Barry M., and Ha Hoang. “Sunk costs in the NBA: Why draft order affects playing time and survival in professional basketball.” Administrative Science Quarterly (1995): 474-494.

If Tebow had been drafted higher, he wouldn’t necessarily be better, but the owner/manager would have stuck with him longer. Too much of a sunk cost.

Postscript 4: Another reader writes:

One thing someone pointed out to me and I can’t get past now: 538 follows the web tradition of just using hyperlinks for references. That format makes sense for generic blogs, but for a page that has a footnote format, why not use footnotes to give the reader a sentence on what the linked piece actually did so they know whether they should click it?