I remember seeing a version of the above in High School. My favourite entries, which I quote to this day, are
"... accidentally strained during mounting" --> "... dropped on the floor"
"... handled with extreme care throughout the experiments" --> "... not dropped on the floor"
and
"correct within an order of magnitude" --> "wrong"
From Futility Closet. Spotted via Boing Boing.
A handy summary prepared by Jesse Sigal. Thanks, Jesse!
Advice
- Determine what is relevant for you to actually benchmark (areas include accuracy, computational complexity, speed, memory usage, average/best/worst case, power usage, degree of achievable parallelism, probability of failure, clock time, performance vs time for anytime algorithms).
- Make sure you run on appropriate data, including generating random (but representable) data and running statistical analysis.
- Consider using multiple datasets and cross-validation.
- Consider the extreme cases as well.- Find benchmarks the field will care about.
Books
- “Writing for Computer Science” by Justin Zobel
- “The art of computer systems performance analysis” (1990) by Raj Jain
Papers
- A. Crapé and L. Eeckhout, “A Rigorous Benchmarking and Performance Analysis Methodology for Python Workloads,” 2020 IEEE International Symposium on Workload Characterization (IISWC), Beijing, China, 2020, pp. 83-93, doi: 10.1109/IISWC50251.2020.00017.
- A. Georges, D. Buytaert, L. Eechkout, “Statistically rigorous java performance evaluation,” OOPSLA '07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems, languages and applications, October 2007 Pages https://doi.org/10.1145/1297027.1297033
- Benchmarking Crimes: An Emerging Threat in Systems Security. van der Kouwe, E.; Andriesse, D.; Bos, H.; Giuffrida, C.; and Heiser, G. Technical Report arXiv preprint arXiv:1801.02381, January 2018.
- Hoefler, Torsten, and Roberto Belli. "Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results." Proceedings of the international conference for high performance computing, networking, storage and analysis. 2015.
- Hunold, Sascha, and Alexandra Carpen-Amarie. "Reproducible MPI benchmarking is still not as easy as you think." IEEE Transactions on Parallel and Distributed Systems 27.12 (2016): 3617-3630.
Online resources
- http://gernot-heiser.org/benchmarking-crimes.html
- https://www.sigplan.org/Resources/EmpiricalEvaluation/
- https://www.acm.org/publications/policies/artifact-review-and-badging-current
A fascinating blog post by Adam Mastroianni, suggesting that peer review is a failed experiment.
From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.
(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)
That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.
Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.
The results are in. It failed.
Thanks to Scott Delman for the pointer.
The post also cites a scientific paper by Mastroianni that he published direct to his blog, circumventing peer review while allowing him to write in a far more readable style. It's a great read, and you can find it here: Things Could be Better.
Via a tweet, the PLDI steering community is asking whether PLDI should join PACMPL. Have your say! (My vote is yes.)
Should @pldi join ICFP, OOPSLA and POPL in publishing its proceedings in the PACM-PL journal? The PLDI Steering Committee would appreciate your views. Please complete this short survey before the end of Thursday 16 June (AoE): forms.office.com/r/HjwYvq1CGw
A master class from Patrick Winston of MIT on how to present ideas clearly. Chockfull of useful advice, much of which I've not seen elsewhere. Recommended.
Reorienting our focus from the hero model’s “big people” to the consideration of big questions will address many of the challenges plaguing universities today: incremental, derivative, low-risk science; faltering funding; relentless focus on quantity of publication; irreproducible research; ongoing complaints of harassment; lack of diversity; an atmosphere that leaves students struggling with mental health; and (despite enormous funding outlays) an inadequately trained workforce in the STEM fields of science, technology, engineering, and math.
We propose that SIGPLAN form a Committee on Conference Data. The committee would be made up of: one organizing-committee representative from each of the flagship SIGPLAN conferences, one early career representative, and, crucially, a professional data collection specialist hired by SIGPLAN. The group would identify and collect key data that is pertinent to conference organization, especially with respect to physical versus virtual conference formats. The committee would make data-driven recommendations to SIGPLAN organizers based on the collected data and guided by core tenets such as community building, inclusivity, research dissemination, and climate responsibility. We realize that this is not a small request, but we are confident that it is both necessary and achievable. If the committee were to form by May 1, 2021, it would be able to start collecting data at PLDI 2021 and continue through the next two years, providing enormous clarity for SIGPLAN organizers at a time when so much is unclear.
Marco Patrignani explains why and how to use colour in your technical papers. The guidelines for making highlighting useful even for colour-blind folk are particularly helpful.
By my colleagues at the University of Glasgow, Muffy Calder and Quintin Cutts. Stunning application of animation and poetry. Text with links here.
Sue Fletcher-Watson describes her plan for A Year of Radical No's and follows up with Nine Months of Saying No – an update. Thanks to Vashti Galpin for the pointer!
So my main fear was that this Strategic Leadership Course would try to feed me time management tips, taking up 6 precious days of my time, when what I need is just LESS WORK. Thank the lord, far from it. ... One session left a particularly strong impression on me. We spent some focused time considering the work-life balance challenges of another person on the course, culminating in offering them some advice. My advice? Say No, for a whole year, to everything new. Conferences, training, collaborations, journal reviews, student supervisions, the whole lot. Their response? Laughter. None of us could imagine doing such a thing.
Stefan Collini writes in the Guardian.
Then there is the rather less obvious contradiction between consumerism and education. Our higher education system is at present structurally consumerist. Even now, it is not widely understood how revolutionary were the changes introduced in 2010-12 by the coalition government in England and Wales (Scotland wisely followed another course). It wasn’t simply a “rise in fees”. It was a redefinition of universities in terms of a market model. The Office for Students is explicitly a “consumer watchdog”. Consumers are defined by their wants; in exchange for payment they are “entitled” to get what they ask for. ...
Universities are, by a long way, the main centres of research and scholarship in our societies; they curate the greater part of our intellectual and cultural inheritance; they provide by far the best source of disinterested expertise; they select and prepare those who will be the scholars and scientists of the future, and so on. Countries all over the world have found that you cannot fulfil these functions by distributing students and academics across all institutions either uniformly or randomly. Some element of selection and concentration is needed, and that brings with it some element of hierarchy, however unofficial. Explicit differentiation of function among higher education institutions might well be preferable to any pretence that they are all doing the same thing and doing it equally well.
The ideal mathematician’s work is intelligible only to a small group of specialists, numbering a few dozen or at most a few hundred. This group has existed only for a few decades, and there is every possibility that it may become extinct in another few decades. However, the mathematician regards his work as part of the very structure of the world, containing truths which are valid forever, from the beginning of time, even in the most remote corner of the universe.
On average, the pilot conferences’ Q&A sessions generated three times more discussion than takes place at a traditional Q&A. A few sessions generated more than ten or fifteen times more, making clear that, while different from a traditional conference, meaningful personal interaction was not only possible, but in certain respects superior.
“The whole process of revisiting those old letters and making that skirt sort of reminded me that you have to apply to a lot of things to succeed,” she said. “A natural part of the process is to get rejected along the way.” ... "I just searched for 'unfortunately' and 'we regret to inform you' in my email"Spotted via Boing Boing.
