By Rachel Gordon
Billions of users routinely flock to the online, anonymously editable encyclopedia knowledge bank for just about everything, but how this unauthoritative source influences our discourse and decisions is hard to reliably trace. Can we measure how living in a Wiki World is playing out in reality?
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Maynooth University, Ireland came up with a friendly stress test: creating new legal Wikipedia articles to examine how they affect the legal decisions of judges.
They set off by developing over 150 new Wikipedia articles on Irish Supreme Court decisions, written by law students, half of which were randomly chosen to be uploaded where they could be used by judges, clerks, lawyers, and so on—the “treatment” group. The other half were kept offline, and this second group of cases provided the counterfactual basis of what would happen to a case absent a Wikipedia article about it (the “control”).
They then looked at two measures—whether the cases were more likely to be cited as precedents by subsequent judicial decisions, and whether the argumentation in court judgments echoed the linguistic content of the new Wikipedia pages.
It turned out the influx of articles tipped the scales: getting a Wikipedia article increased a case’s citations by more than 20%. The increase was statistically significant and the effect was particularly strong for cases that supported the argument the citing judge was making in their decision (but not the converse). Unsurprisingly, the increase was bigger for citations by lower courts—the High Court—and mostly absent for citations by appellate courts—the Supreme Court and Court of Appeal. The researchers suspect that this is showing that Wikipedia is used more by judges or clerks who have a heavier workload, for whom the convenience of Wikipedia offers a greater attraction.
Their statistical model essentially compared how much citation behavior changed for the treatment group (first difference: before vs. after) and how that compared with the change that happened for the control group (second difference: treatment vs. control).
“To our knowledge, this is the first randomized field experiment that investigates the influence of legal sources on judicial behavior. And because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation,” says MIT researcher Neil Thompson, the lead author of the research. “The fact that we wrote up all these cases, but the only ones that ended up on Wikipedia were those that won the proverbial “coin flip,” allows us to show that Wikipedia is influencing both what judges cite and how they write up their decisions. Our results also highlight an important public policy issue. With a source that is as widely used as Wikipedia, we want to make sure we are building institutions to ensure that the information is of the highest quality. The finding that judges or their staffs are using Wikipedia is a much bigger worry if the information they find there isn’t reliable.”
The paper was written by Thompson, and Flanagan, Edana Richardson and McKenzie of Maynooth University, and Xueyun Luo of Cornell University. The paper will be published in: Kevin Tobia (ed). The Cambridge Handbook of Experimental Jurisprudence. New York: Cambridge University Press.
AR #112
“Wikipedia and the Slant Factor”
by Martin Ruggles
