Wikipedia, the early years.

January 2001. Just twelve months after dodging the “ravages” of the millennium bug, another devastating event was taking place: the launch of Wikipedia by Jimmy Wales and Larry Sanger.

Wikipedia front page on March 31, 2021

The basic premise seemed innocuous enough: write a complete encyclopedia from scratch, collaboratively. The project met polite skepticism, then fierce derision, and then, against all odds, it became the closest thing the internet has to a factual baseline.

The first shot across the bow came in the form of Péter Jacsó’s review in Online magazine, where he likened Wikipedia to a prank or joke and mocked its 100,000-article goal as twice the Columbia Encyclopedia, a single-volume encyclopedia whose most recent edition was published in 2000.

The main wave of attacks to the “communal encyclopedia of the people, by the people, and for the people, which shall not perish from the earth” would come in 2004. Al Fasoldt in the Syracuse Post-Standard would continue to critique the process: “Wikipedia is a do-it-yourself encyclopedia, without any credentials.” Robert McHenry, formerly of Encyclopedia Britannica, would call Wikipedia “The Faith-Based Encyclopedia.”, where every article stays “forever open to the uninformed or semiliterate meddler.”

The attacks would also come from inside the house. In 2004, Larry Sanger, one of the co-founders of Wikipedia, would say: “Wikipedia lacks the habit or tradition of respect for expertise.” He demanded expert review and predicted a fork that never materialized. Nicholas Carr sampled two random entries and concluded the site is unreliable at the factual level, with often appalling writing. Jimmy Wales joined him, acknowledging real quality problems.

Wikipedia’s editability would also create problems by taking hoaxes to a whole new level. In 2005, a hoax bio of John Seigenthaler, an American Journalist, connected Seigenthaler to the Kennedy assassinations and stayed up for 132 days.

The tide of attacks wouldn’t turn, and Wikipedia remained a credible source until Nature magazine went beyond rhetoric and ran an empirical test. Compare 42 science articles of Wikipedia and Encyclopedia Britannica and evaluate their quality. The end result: 162 errors in Wikipedia vs 123 in Britannica… and the rest is history.

The unruly, uncredentialed, unreliable Wikipedia, the very project that experts dismissed as a vandal’s playground, now serves as the gold standard. Wikipedia spawned a whole new branch of social and information science. Today, AI systems are trained on Wikipedia, and Hallucination detectors check outputs against its content.

18 years later…

November 30, 2022. A transformer-based large language model wrapped in a deceptively simple chatbot interface was announced by Sam Altman from OpenAI as a cool demo:

ChatGPT gave the public something it wasn’t expecting: a fluent, free source for instant answers. Like Wikipedia before it, ChatGPT offered authoritative-sounding responses on virtually any topic… and it was immediately accused of the same crime: producing confident falsehoods indistinguishable from the truth.

chatgpt free research previewChatGPT on Dec 2022

This time the backlash was much faster. Within days, ChatGPT had been banned from Stack Overflow due to “The average rate of getting correct answers from ChatGPT is too low”, with answers that look authentic and cost nothing to produce. Ted Chiang in The New Yorker framed it as: “Think of ChatGPT as a blurry JPEG of all the text on the Web”.

Chomsky, Roberts, and Watumull echoed the no-credentials argument, calling the model “a lumbering statistical engine for pattern matching”. Gary Marcus, a researcher at NYU, called the systems “unreliable and potentially dangerous” ten days after launch. His opinion has not improved much since.

Where Wikipedia had purposeful hoaxes, ChatGPT has hallucinations. In Mata v. Avianca, Mr. Mata’s lawyers used ChatGPT to generate a legal brief, and ChatGPT hallucinated entire precedents and case histories. Similarly, CNET tried using an “AI Engine” to generate news articles, resulting in glaring errors and a wave of bad publicity.

Fortunately, lessons were learned quickly. The Avianca and CNET incidents produced verification norms, court rules on AI filings, and grounding tooling. The fixes took the form of a procedure tacked on to handle the system’s inherent limitations.

LLMs are inherently different

Wikipedia is centralized and transparent. Wikipedia pages have unique URLs with an attached edit history. Changes can be made for free, by anyone, and whenever a change is made, everyone immediately has access to it. New information propagates quickly. If an error is introduced, you can undo or correct the change as needed.

Language models are very different beasts. Answers are generated anew with each interaction based on the values of billions or trillions of numerical values. Thtere’s no way to assign a specific meaning to any specific number and no way to know how to modify them to correct a mistake. Corrections and changes can only be introduced by retraining the model from scratch, with an associated price tag in the order of tens or hundreds of millions of dollars.

Wikipedia’s human editors have an intrinsic bias towards truth, while models are indifferent to the truth of the outputs they generate. They produce “bullshit”, in the technical, philosophical sense. Crowds of editors are amenable to rules, regulations, social pressure, and procedures in a way that language models never will be.

Despite these limitations, recent progress in harnesses, Retrieval Augmented Generation (RAG), new architectures, and training procedures has driven tremendous gains across a wide range of benchmarks, including proving mathematical theorems, controlling robots, medicine, software engineering, etc.

Despite the initial hiccups, Wikipedia evolved into a trusted source of information for millions of people worldwide and is now one of the reference datasets frontier models are trained on, and the ground truth LLMs are graded against. LLMs appear to be on a similar, if much faster, trajectory, with hallucination rates steadily decreasing with each new generation of models.

The Nature study ended the mistrust of Wikipedia by putting a number on the gap. Distrust of LLMs will likely end the same way, with quantitative studies and practical evaluations. The difference is that this time you don’t have to wait for Nature; you can run the study yourself, on your own system, before your users find the errors for you.

That’s what I teach in my live workshop, Build a Production-Grade LLM Eval Harness. In four hours, you build a working harness with rubrics, LLM-as-judge scoring, paired statistical tests, and CI integration, all on open-source tooling. You leave with running code, not slides. It’s built for AI engineers, tech leads, and data scientists shipping LLM features who need proof, not vibes.

Join the next cohort and use the code MEASURE20 at checkout for a discount.

Stop arguing about what the models are. Start measuring what they do.