An inside look at the Sloan Sports Analytics Conference research paper contest

(Update: Click here for Part II, as Sloan updated its ticket policy and is now awarding all poster-winners 1 free ticket)

The Sloan Sports Analytics Conference (SSAC) has sold out every year since its 2006 inception, garnering the attention of ESPN, the New York Times, Time Magazine, and countless other worldwide media outlets, while promoting, according to its website, “the increasing role of analytics in the global sports industry.” This year, the conference will be held in Boston on February 28 and March 1.

One of the most academic portions of SSAC is its research paper contest (RP), which begins each year in September with an abstract submission, and, for worthy candidates, ends with a lengthy paper due in January.  This year, I was one of a likely several dozen finalists (organizers did not offer the actual number of paper submissions) whose initial abstract was accepted into a RP submission form.

Before continuing, I should probably clarify what’s at stake in the RP contest. First place in the competition comes with a $20,000 reward, and the second prize is $10,000. In my case as a graduate student, and for likely a few other participants, that would roughly double my salary.

Further, each selected finalist earns a free ticket to SSAC, currently $575 plus a $26 registration fee, and a presentation, which, given the expansive and distinguished audience, is unprecedented exposure (side note – who pays $600 for a two day conference!!!).

At the beginning of January, my six-page paper (that was the limit), with three pages of an appendix, was officially submitted. Here’s what happened next.

Saturday, February 1, 8:24 pm

Myself and the other finalists receive this email from the SSAC research team, stating

Dear Research Paper Participant,

As we approach our selection for finalists, we would love to be able to feature papers for publication and publicity immediately upon selection. In order to do so, we will need you to reformat your submission to the template attached. If you could kindly resubmit your paper by Monday 2/3 at 12:00 midnight EST as a doc file to researchpapers@sloansportsconference.com, that would allow us to feature your paper immediately if selected.

We greatly appreciate accommodating the situation and look forward to receiving your submission.

Thanks,
The Research Paper Team

Attached to the email was a research paper template, including advertisements from Ticketmaster and a company called 42 Analytics.

At this point, each finalist had to have been feeling great. I know I was. I mean, they wouldn’t have you reformat your paper using the conference’s template and then not accept it, right?

Also, all RP finalists were left to assume that they were still in the running for a presentation and the grand prize – no mention of alternative ‘prizes’ had been suggested yet.

Sunday, February 2, all day

If you are still reading, you understand why I spent 4-5 hours reformatting my submission on Sunday, making sure each graph and table looked as perfect as I could make them. I assumed that the other finalists were doing the same thing, and that the SSAC was smart enough to not to have emailed any RP’s which were not going to get published.

Again, it’s a Sunday, with roughly 36 hours left before the artificial paper deadline that all of the finalists faced.

Tuesday, February 4, 1:17 pm

I received an email, with the first part of the email stating, “We would like to congratulate you.” This has to be a good thing, right?

Here’s what the SSAC research team wrote:

Dear Researcher,

We would like to congratulate you on your acceptance to display a poster of your research at the 8th Annual MIT Sloan Sports Analytics Conference on February 28 and March 1. In addition, we would like to publish your paper on our website for publication to the general public. Due to stiff competition and time constraints, we will not be able to include a presentation of your research during the event in Boston. However, you are welcome to come to present your research with your poster but unfortunately we cannot provide complimentary admission. Please read this email in its entirety as it includes many important details

 ….

Please feel free to contact us with any questions you may have. We look forward to seeing you at the conference!

Regards,
Research Paper Team

While not perfect, this email didn’t appear to be the worst thing in the world on my end – I got to present a poster at a prestigious sports analytics conference. But I’ll get to my story next.

Meanwhile, as I debated the design of my poster, I heard an account of one of the other researchers, who, like me, submitted a paper. After reformatting just like I did, this paper was completely rejected!

Said my contact, an academic at a distinguished US institution: “I spent hours reformatting the paper. And it was a Sunday. Two days later, I realize my efforts were a complete waste of time.”

Who does that? Who forces people spend countless hours of unnecessary work, only to then say you don’t want what was produced? At best it’s unprofessional.

But I’m just getting started.

Tuesday, February 4, 1:17 pm

Personally, I still had a lot on my mind. For starters, I was not yet aware that my paper, accepted as a poster, had been eliminated from prize money. Further, student tickets ($200) were all sold out. So how was my poster presentation going to work? (Note: Small mistake on my end here: all tickets were sold out, not just the student rate tickets. Eventually I figured this out.)

Hi ***** (name omitted),

A few quick questions. First, does poster relegation indicate that the paper does not qualify for the research prize?  Second, student rate tickets for the conference are sold out. Will I have to pay full price to present the poster?  …

Thanks

Mike

 Tuesday, February 4, 1:43 pm

Hi Michael,

Sorry, unfortunately poster relegation does mean that you are no longer in the running for the research prize. If you can get your hands on a ticket to present, please do so. We would love for you to present your poster to those interested at the conference.

Unfortunately I cannot disclose the total number of submissions this year. …

Regards,

****

A few points here. First, SSAC agreed with my term “poster relegation.” Posters are not to be congratulated on – instead, they are relegated, appearing to be another way for SSAC to make money, seeing as I’d have to pay to attend.

Second, there’s this: “If you can get your hands on a ticket, please do so. We would love for you to present your poster to those interested.”

Wait – you want me to present my poster? Aww shucks!!! I’m so honored that I can present my own poster. Here I was thinking that Herm Edwards was going to discuss my generalized linear mixed effects models, or that Jeanie Buss would explore my graphs in ggplot2. It would actually get to be me at my own poster?

Okay, then.  Now all I needed was this thing called a ticket.

Tuesday, February 4, 2:03 pm

Hi ****,

… all tickets are currently unavailable. So this means that even though I have a spot to present a poster, I might not be able to actually attend myself?

…

Mike

Tuesday, February 4, 2:17 pm

 Hi Michael,

 …I can talk to our ticketing team to see if any exceptions can be made although it is unlikely.

Thanks,

****

Thus, it appeared that even though my poster will be there, and that my paper would be published on the Sloan website, I would be unable attend the conference myself, either to back up my paper or to explain it. I can’t even pay the full $575 registration fee (plus $26 service) to attend!

Like some of the other poster finalists probably did, I felt used.  SSAC took my research as a form of showing-off (look at this, we really do value research here!), without it actually impacting their bottom line. The conference is not about advancing research – if it was, it could just highlight the best academic papers from, for example, the Journal of Quantitative Analysis in Sports or the Journal of Sports Economics.

SSAC is a business.  In fact, it’s a business that Fast Company called the third most innovative sports company in 2012. Of course, as a business, Sloan’s research contest is run by business people (including MBA students from MIT). You’ll see how that manifests itself when I show you the eight papers which were named finalists.

Wednesday, February 4, afternoon

The eight presentation-winning papers are released. You can read them here (find the links using the preface 2014_SSAC). Of course, while it’s too early to explore every detail of the papers, a few things stood out.

1 – Reproducibility, or lack thereof:

By my estimation, six of the eight data sets used are propriety. Sloan is not sponsoring a research conference as much as it’s sponsoring a data access contest. Kirk Goldsberry, a writer at ESPN and a finalist in 2014, more or less admitted as much in a column on Grantland, writing,

 …I received a call from Brian Kopp, the John the Baptist of player-tracking data in the NBA and the person in charge of the SportVU project at STATS LLC in Chicago. I was working at Harvard at the time, and Kopp was offering to share his incredible data set with basketball-minded academics; he asked me if I wanted to “play with some optical tracking data.” I leapt at the chance…

So why does this matter?  It means that six of the eight papers are not reproducible. Specifically, the best statisticians and sports researchers in the country could not know for sure if the paper’s findings are correct, simply because the data is private. Who double checks SportVU’s data? Who can double check Goldsberry and his colleagues’ work?  The answer is no one.

Reproducibility is a buzzword in the academic community, in large part because it has been suggested that many published findings are false.  Johns Hopkins biostatistics professor Roger Peng offers a nice summary here, stating, “I think the key question we want to answer when seeing the results of any data analysis is ‘Can I trust this analysis?’”

With most of the final papers at Sloan, we have no way of trusting the results.

2 – Connections

A quick Google search suggests that five of the final eight papers have email addresses attached to ESPN, Harvard, MIT, or Disney (and in some cases, both), and that one of the other three finalists includes an MIT graduate.

Harvard and MIT, of course, have helped run the conference since its inception (Sloan is the MIT business school, after all), and ESPN, which is owned by Disney, is the sponsor’s primary sponsor.

Even better, the conference was founded by Jessica Gelman and Daryl Morey. Where did they go to business school? Yup, Harvard (Gelman) and MIT (Morey).

Some might consider this set-up a conflict of interest. For what its worth, half of the eight RP finalists in 2013 were affiliated with Harvard or MIT (described here).

Still, there’s more.

Thursday, February 6, morning

Goldsberry’s column (which I referenced earlier) comes up on Grantland. For the most part, I really enjoyed the article. Goldsberry does an outstanding job of simplifying his work and explaining it, and his graphics are outstanding. Further, Goldsberry’s top statistics guy, Harvard’s Luke Bornn, has an outstanding academic publication record. While I referenced reproducibility earlier, Bornn’s background gives me a high degree of confidence in the metrics employed in this paper.

But something’s still fishy. Goldsberry posted the article less than 48 hours after the research finalists had been announced. And not just an short blog post, but a 3,345 word exploration of NBA data, complete with one drawing, three graphs, and a video attachment. And he writes:

This unique approach to basketball analysis is the subject of a research paper to be presented later this month at the 2014 MIT Sloan Sports Analytics Conference in Boston.

Given the two-day turnaround between Tuesday and Thursday, and, more importantly, the intense scrutiny Grantland’s editors apparently pour into every article (cough, cough), Goldsberry had to have been working on this article before research paper finalists were announced.  Even further, would Grantland have run an article on an RP which was not eligible to win the research prize or to present at the conference ?

Can you imagine the alternatives?  “This unique approach to basketball analysis is the subject of a poster…” 

Of course not.

While Goldsberry and company move on, my story, sadly, appears to be over: the conference hosting my poster can’t get me a ticket to present it.

Thursday, February 6, 1:45 pm

Hi Mike,

If you are not going to attend, we still encourage you to make a poster so that it can be displayed at the conference for everyone to see. It is a great way to get your research seen by whoever attends the conference. Please let me know what you decide.

Thanks,

*****

 ************************************************************************************

 Moving forward, here are five easy ways for Sloan to improve its RP contest:

1- Value reproducibility.

Specifically, appreciate and award submissions which store code and/or data on, among other locations, Github pages or personal websites. This increases accountability, and provides tools for researchers to learn new material and build on existing work. We hear the term ‘Big Data’ thrown a lot, and, in medical fields, privacy of ‘Big Data’ is admittedly a huge issue. With sports, however, no one entity owns the data. Even more, can it really be ‘Big Data’ if only a select few have access to it?

2-Increase judging expertise

Topics in the 2014 research paper finalists touched on the fields of data mining, psychology, and economics.  Experts from these areas are needed in order to truly understand the importance of these works.

For example, I found two quick examples of finalists that took a tone that they were introducing new material, when in fact they were most certainly not.

First, a RP on the most valuable commodities in sport (“Can’t buy much love…”) encouraged this tweet from University of Michigan sports economist Rodney Fort

Ignoring important recent research into sports analytics seems like a sub-optimal way of advancing the field.

But you don’t even need Rodney Fort to do a background check. In fact, Google Scholar will do.

A second paper looks at the existence of the “hot-hand” fallacy (“New approach…”), writing

“Today, among the academic crowd, the Hot Hand is almost universally considered a “fallacy.””

That’s just not the case.

Three of the most recent peer-reviewed papers on the subject, including Arkes (2010), Stone (2011), and Shea (2013), argue in favor of the hot-hand, all doing so through the lens of basketball. Not one of these papers was cited by the “New approach” cohort.

Of course, this could be the writer’s omitting the references due to a lack of space. A more pessimistic argument is that the RP’s purposefully ignored prior research in order to further their own causes.

The “New approach” to the hot-hand paper cited six peer reviewed papers in its introduction, all of them written prior to 2010.

3- Clarify standards

Make standards clear to those submitting papers what they are getting themselves into before papers are selected, not afterwards. This seems pretty obvious, particularly for a conference run by MIT graduate students.

4- Blind the research submissions.

Nearly every research paper contest that I’m familiar with instructs authors to not include their name and affiliation with their submission, or to include such information separately. Pretty sure the reasons for doing this are obvious.

5- Set a word count (and a figure count), not a page count.

Given the 6-page limit I could manage only 2600 words into my paper. Another finalist fit 4,600 into the same space. Still not sure how that happened.

To conclude, hopefully my experience helps someone considering a submission for future contests.

In any case, if you do make it to the 2014 Sloan conference, be sure to check out my poster.

I look forward to hearing about how it looked.

(Update: Click here for Part II)