Wadler's Blog

Conference Reviewing Considered Harmful offers a delightful and provocative view on the strengths and weaknesses of conferences, and how they might evolve.

This paper develops a model of computer systems research to help prospective authors understand the often obscure workings of conference program committees. We present data to show that the variability between reviewers is often the dominant factor as to whether a paper is accepted. We argue that paper merit is likely to be zipf distributed, making it inherently difficult for program committees to distinguish between most papers. We use game theory to show that with noisy reviews and zipf merit, authors have an incentive to submit papers too early and too often. These factors make conference reviewing, and systems research as a whole, less efficient and less effective. We describe some recent changes in conference design to address these issues, and we suggest some further potential improvements.

The paper applies the methods of the systems community to evaluate systems conferences. Three cheers for reflexive research!