Folks who have submitted abstracts over the past two years to the Sloan Sports Analytics Conference research paper contest were recently surveyed as to their thoughts on the contest.
Here are my (expanded) answers to the open ended question “Do you have any other suggestions or comments that will help us improve the research papers competition?”
1- Maintain a strong prize pool, but eliminate the crazy discrepancy between 1st and, say, 5th place.
In the current set-up, first place is $20k, second place $10k, and third place onwards is nothing. This structure incentivizes researchers to oversell their findings, because admitting that your work is simply building on the research of others is not nearly as sexy as claiming to be the first in your field to find something.
What’s a more equitable system? One that encourages good content, appropriate citation of sources, and makes it clear why each paper is relevant to advancing sports analytics.
From a prize perspective, this makes it less of a crapshoot. Financially, each finalist gets $2k and a free ticket. Winner get $10k. Boom, done.
2- Reward participants whose submission is reproducible.
I cannot remember a single finalist paper that has either included (i) its data or (ii) its source code. This is not a good (note: I’m also guilty. I didn’t submit code or data two years ago). Given that the majority of findings in professional research are not reproducible, it is difficult – perhaps impossible – to know if each paper truly got things right. Rewarding papers that include data and source code would be a major step in promoting reproducible research.
Of course, work is only reproducible if the data set is public. A more aggressive but related idea would be to use separate tracks for both proprietary and public data. This was suggested a year ago by analyst Christopher Long (and perhaps by others). Such a distinction levels the playing field among researchers who have good work to share but are working with standard data, where it is becoming more and more difficult to make novel discoveries each year.
3- Implement a conference proceedings section.
For many people in academics, there is a lesser incentive for submitting to Sloan given that, unless your paper finishes as one of the finalists, all of your work is for not. Having a conference proceedings would likely encourage more submissions in this regard. If you are worried about the cost, publish online only and charge anyone who wants a hard copy. This would be very cheap.
4- Also allow submissions in TeX.
For years, the conference has used the same Microsoft Word template for participants. But many analytics researchers use TeX and only TeX for their work, as the formatting, particularly for mathematical notation, is substantially easier and more readable in TeX than in Word. TeX is also more visually appealing than Word.
Allowing submissions in both TeX and Word seems like an easy compromise.
**********
Happy to hear other takes as well. I am appreciate of the fact that SSAC has upgraded the rewards for poster recipients over the past few years. Futher, the fact that SSAC has implemented a survey in the first place is hopefully a promising sign of changes to come.
StatsbyLopez 