16.5.23
Naomi Klein on AI Hallucinations
Amongst all the nonsense, something sensible in the press about AI: "AI machines aren’t ‘hallucinating’, But their makers are" in The Guardian. Written by Naomi Klein, the author of one of my favourite books, This Changes Everything.
But first, it’s helpful to think about the purpose the utopian hallucinations about AI are serving. What work are these benevolent stories doing in the culture as we encounter these strange new tools? Here is one hypothesis: they are the powerful and enticing cover stories for what may turn out to be the largest and most consequential theft in human history. Because what we are witnessing is the wealthiest companies in history (Microsoft, Apple, Google, Meta, Amazon …) unilaterally seizing the sum total of human knowledge that exists in digital, scrapable form and walling it off inside proprietary products, many of which will take direct aim at the humans whose lifetime of labor trained the machines without giving permission or consent.
This should not be legal. In the case of copyrighted material that we now know trained the models (including this newspaper), various lawsuits have been filed that will argue this was clearly illegal. Why, for instance, should a for-profit company be permitted to feed the paintings, drawings and photographs of living artists into a program like Stable Diffusion or Dall-E 2 so it can then be used to generate doppelganger versions of those very artists’ work, with the benefits flowing to everyone but the artists themselves?
The painter and illustrator Molly Crabapple is helping lead a movement of artists challenging this theft. “AI art generators are trained on enormous datasets, containing millions upon millions of copyrighted images, harvested without their creator’s knowledge, let alone compensation or consent. This is effectively the greatest art heist in history. Perpetrated by respectable-seeming corporate entities backed by Silicon Valley venture capital. It’s daylight robbery,” a new open letter she co-drafted states.
The trick, of course, is that Silicon Valley routinely calls theft “disruption” – and too often gets away with it. We know this move: charge ahead into lawless territory; claim the old rules don’t apply to your new tech; scream that regulation will only help China – all while you get your facts solidly on the ground. By the time we all get over the novelty of these new toys and start taking stock of the social, political and economic wreckage, the tech is already so ubiquitous that the courts and policymakers throw up their hands.
We saw it with Google’s book and art scanning. With Musk’s space colonization. With Uber’s assault on the taxi industry. With Airbnb’s attack on the rental market. With Facebook’s promiscuity with our data. Don’t ask for permission, the disruptors like to say, ask for forgiveness. (And lubricate the asks with generous campaign contributions.)
Labels: AI, Computing, Politics
24.3.23
Benchmarking best practices
A handy summary prepared by Jesse Sigal. Thanks, Jesse!
Advice
- Determine what is relevant for you to actually benchmark (areas include accuracy, computational complexity, speed, memory usage, average/best/worst case, power usage, degree of achievable parallelism, probability of failure, clock time, performance vs time for anytime algorithms).
- Make sure you run on appropriate data, including generating random (but representable) data and running statistical analysis.
- Consider using multiple datasets and cross-validation.
- Consider the extreme cases as well.- Find benchmarks the field will care about.
Books
- “Writing for Computer Science” by Justin Zobel
- “The art of computer systems performance analysis” (1990) by Raj Jain
Papers
- A. Crapé and L. Eeckhout, “A Rigorous Benchmarking and Performance Analysis Methodology for Python Workloads,” 2020 IEEE International Symposium on Workload Characterization (IISWC), Beijing, China, 2020, pp. 83-93, doi: 10.1109/IISWC50251.2020.00017.
- A. Georges, D. Buytaert, L. Eechkout, “Statistically rigorous java performance evaluation,” OOPSLA '07: Proceedings of the 22nd annual ACM SIGPLAN conference on Object-oriented programming systems, languages and applications, October 2007 Pages https://doi.org/10.1145/1297027.1297033
- Benchmarking Crimes: An Emerging Threat in Systems Security. van der Kouwe, E.; Andriesse, D.; Bos, H.; Giuffrida, C.; and Heiser, G. Technical Report arXiv preprint arXiv:1801.02381, January 2018.
- Hoefler, Torsten, and Roberto Belli. "Scientific benchmarking of parallel computing systems: twelve ways to tell the masses when reporting performance results." Proceedings of the international conference for high performance computing, networking, storage and analysis. 2015.
- Hunold, Sascha, and Alexandra Carpen-Amarie. "Reproducible MPI benchmarking is still not as easy as you think." IEEE Transactions on Parallel and Distributed Systems 27.12 (2016): 3617-3630.
Online resources
- http://gernot-heiser.org/benchmarking-crimes.html
- https://www.sigplan.org/Resources/EmpiricalEvaluation/
- https://www.acm.org/publications/policies/artifact-review-and-badging-current
Labels: Academia, Computing, Programming Languages
7.3.23
Benchmarking Crimes
- Benchmarking Crimes, by Gernot Heiser.
- Empirical Evaluation Guidelines, from SIGPLAN.
Labels: Academia, Programming Languages
15.12.22
The Rise and Fall of Peer Review
A fascinating blog post by Adam Mastroianni, suggesting that peer review is a failed experiment.
From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church. Scientific journals appeared in the 1600s, but they operated more like magazines or newsletters, and their processes of picking articles ranged from “we print whatever we get” to “the editor asks his friend what he thinks” to “the whole society votes.” Sometimes journals couldn’t get enough papers to publish, so editors had to go around begging their friends to submit manuscripts, or fill the space themselves. Scientific publishing remained a hodgepodge for centuries.
(Only one of Einstein’s papers was ever peer-reviewed, by the way, and he was so surprised and upset that he published his paper in a different journal instead.)
That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals. That funding turned into a deluge of papers, and journals that previously struggled to fill their pages now struggled to pick which articles to print. Reviewing papers before publication, which was “quite rare” until the 1960s, became much more common. Then it became universal.
Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected. You can still write to your friends about your findings, but hiring committees and grant agencies act as if the only science that exists is the stuff published in peer-reviewed journals. This is the grand experiment we’ve been running for six decades.
The results are in. It failed.
Thanks to Scott Delman for the pointer.
The post also cites a scientific paper by Mastroianni that he published direct to his blog, circumventing peer review while allowing him to write in a far more readable style. It's a great read, and you can find it here: Things Could be Better.
12.11.22
IO Scotfest: The Age of Voltaire - Nov 18-19
IOHK/IOG will be hosting a meeting at Edinburgh next week. Available online, plus an in-person meetup for folk near Edinburgh.
Let’s celebrate the dawning of a new era for #Cardano together. Join us for a virtual event that will showcase the community’s achievements over the last 5 years & discuss IOG’s vision for the future of Cardano. Learn more: https://lnkd.in/g2bzZEtR
Labels: Cryptocurrency, IOHK
26.9.22
Angry Reviewer
26.8.22
Help, please! Do you know any applications of my work?
When writing an application, it sometimes help if I can point out that monads and type classes, which my research contributed to, are used to process every post on Facebook. (Via Haxl. Thanks, Simon Marlow!)
Do you know of other applications of my work? If so, please email me or list them in the comments. (You can find my email at the bottom of my home page.)
Possible example: I gather Twitter uses monads and implicits in Scala (where implicits were influenced by type classes), but it's hard to find confirmation online. Do you know whether they are used, and how heavily? (It's easier to find such confirmation for The Guardian.)
Possible example: Do you make heavy use of generics in Java? I contributed to their design.
Possible example: I gather protocols in Swift are in part inspired by type classes, but it is hard to find confirmation online. Can you point me to confirmation?
There are many other possibilities. I hope you know some I haven't dreamed of!
Many thanks for your help. Answers are welcome at any time, but would be most useful if they can be provided by 2 September 2022.
Labels: F#, Functional Programming, Gender, Haskell, Java, JavaScript, Lego, Object-Oriented, Programming Languages, Scala
27.7.22
Loose Ends
My daughter Leora Wadler is directing and producing a play, Loose Ends.
Four strangers from differing classes and backgrounds attempt to drink and make merry on their last night in halls against the backdrop of a missing girl – but each of them are carrying secrets.
Hope Street Theatre, Liverpool, 7.30pm, Thursday 4 and Saturday 6 August.
9.6.22
Should PLDI join PACMPL?
Via a tweet, the PLDI steering community is asking whether PLDI should join PACMPL. Have your say! (My vote is yes.)
Should @pldi join ICFP, OOPSLA and POPL in publishing its proceedings in the PACM-PL journal? The PLDI Steering Committee would appreciate your views. Please complete this short survey before the end of Thursday 16 June (AoE): forms.office.com/r/HjwYvq1CGw
Labels: Academia, ACM, Programming Languages, SIGPLAN
31.5.22
No, No, No
No person would give up even an inch of their estate, and the slightest dispute with a neighbor can mean hell to pay; yet we easily let others encroach on our lives—worse, we often pave the way for those who will take it over. No person hands out their money to passers-by, but to how many do each of us hand out our lives! We’re tight-fisted with property and money, yet think too little of wasting time, the one thing about which we should all be the toughest misers.
Labels: Productivity
