An AI agent just audited the trial that changed the world.

It cannot certify it.

Digger readers will remember the paper-forensics agent — the AI auditor that noticed 259 patients had quietly vanished from a major trial. This week it read the most consequential paper of the century: “Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine” the critical pandemic paper which charted humanity’s course for the next several years. This was the trial that proved to the world that this highly novel mRNA product was 95% effective, paving the way for it to be rolled out en masse.

So what did the paper-forensics audit find when it looked at the original Pfizer trial? Three audits so far have produced twelve flagged findings. The current status: CANNOT CERTIFY.

Audit of the paper on gather.is

Nothing it found was hidden, because each flag traces to documents that have been sitting on public websites since the month of authorisation. Which raises the question this whole piece is really about: given the obvious incentive to massage the data, shouldn’t regulators have applied the same level of scrutiny to Pfizer that paper-forensics has? Pfizer stood to make tens of billions of dollars if one number was within range. A regulator that took its role seriously would have found everything written below in December 2020. So let’s go through what the agent found.

The uncounted sick

What we were told, first. Here’s CDC Director Rochelle Walensky, on national television, 29 March 2021:

“Our data from the CDC today suggests that vaccinated people do not carry the virus, don’t get sick, and that it’s not just in the clinical trials, but it’s also in real-world data.”

I’m picking on Walensky because it’s the most obvious example, but there were dozens of others. The implication, repeated again and again, was to get vaccinated to save grandma. The promise was no cases, no spread, pandemic over — and it all rested on the trial’s headline: 95% efficacy, from 8 confirmed cases in the vaccine arm against 162 on placebo.

So what did the agent find odd about this study?

In the trial, a “case” was not a sick person. It was a sick person with a positive PCR confirmed by the central lab. Isn’t the problem here obvious? Should it have taken an AI agent to pick this out? Let’s state the problem clearly: If “COVID” is whoever was confirmed by a PCR test, then deciding who should and should not be PCR tested is a lever you can pull to influence the case count. The case count is the efficacy number.

Why is this such a big problem? In the FDA’s own briefing document, published the day the advisory committee met, the agent found this buried on page 42:

“Among 3410 total cases of suspected but unconfirmed COVID-19 in the overall study population, 1594 occurred in the vaccine group vs. 1816 in the placebo group.”

What?!

There were 3,410 trial participants who got “suspected but unconfirmed“ COVID. They were never counted, because no one confirmed them as a ‘case’ with PCR. There were twenty times more suspected cases than confirmed ones, split almost evenly between the arms.

So what happens if we add them to the confirmed cases? Efficacy against symptomatic illness collapses from 95% to roughly 19%. The FDA offered a partial explanation — symptoms in the week after injection were probably (possibly) the shot’s own side effects mimicking COVID. Ok, so we can be stricter: strip all of those out and efficacy recovers to... 29%. But the FDA’s threshold for authorising a COVID vaccine was 50%.

A vaccine that’s only 19–29% effective simply wouldn’t have been authorised. It wouldn’t have been rolled out and then mandated all over the planet. The paper itself — the one published in the New England Journal of Medicine — simply never mentions these 3,410 people. There’s no accounting of how many tested negative, how many were never tested, or why participants with symptoms of COVID, on a COVID trial, were never confirmed as COVID cases.

This is crazy.

Peter Doshi, senior editor at The BMJ, published exactly this calculation on 4 January 2021 — “this category of disease cannot be ignored simply because there was no positive PCR test result” — and the response was a shrug.

The world laboured on under the massive cultural illusion this was an effective product, whilst data the regulators had in their hands raised very serious questions about its effectiveness and the trial’s integrity. Five years later, UKHSA’s own formal effectiveness estimates — adjusted & official — put two doses of Pfizer against symptomatic Omicron at “around 15%” effective “25 weeks after the second dose.” The number buried in the trial’s own paperwork looks a lot more like the world we actually lived through than the 95% ever did.

The vanishing 311

What we were told: the trial had 43,500 participants with gold-standard randomisation.

What the agent found: 7,025 of 43,548 randomised participants (around 16%) never made it into the final “primary efficacy” analysis. Most of these exclusions were mundane, but one row of the FDA’s Table 2 is not: participants excluded for “important protocol deviations” — 311 on the vaccine versus 60 on placebo. That’s a five-to-one asymmetry in a judgement call over protocol deviations. Would we reasonably expect to find such an asymmetry in a well randomised trial? A trial where the entire result hangs on a gap of just 154 cases? No explanation appears anywhere in the published record.

If you incorrectly remove a few hundred of the right people from one arm of a trial that’s this finely balanced, you can move the headline finding. Maybe every one of those 311 exclusions was justified, but the agent couldn’t find any way to check, so it flagged it.

The invisible sites

What we were told: rigorously overseen, independently monitored, the most scrutinised trial in history.

What the record shows: in November 2021, The BMJ published a whistleblower account from a contractor running Pfizer trial sites in Texas. According to the BMJ, the company (Ventavia) “falsified data, unblinded patients, employed inadequately trained vaccinators, and was slow to follow up on adverse events reported in Pfizer’s pivotal phase III trial.” Ventavia enrolled over 1,000 participants. The FDA inspected 9 of the trial’s 153 sites but Ventavia’s trial sites weren’t among them.

Having steered the agent towards these allegations first (give that a go if you’re using it), the agent was alarmed to find that across all five published trial documents there is no site-level breakdown of anything. Enrolment, cases, deviations, everything is pooled globally, which means allegations like the ones published in the BMJ can’t actually be checked against the data. The agent decided this was a very serious problem: allegations of this nature were raised, they were never resolved, and we have no way to verify them. It considered this a serious integrity issue.

The poster number

What we were told: 95% effective.

What the agent computed: that’s relative risk reduction. In absolute terms, a participant’s risk of confirmed COVID fell from 0.88% to 0.04% over the trial’s two-month window, which is about 0.84 percentage points. This means (as is now widely known) the absolute efficacy of that product - if you take all of their numbers at face value - is 0.84. You’d need 119 people vaccinated before you prevented one confirmed case.

The stopped clock

Even the trial's two-month window was a choice. The paper reported safety and efficacy over "a median of 2 months" of follow-up, which is the minimum the FDA would accept for emergency authorisation. The pre-registered protocol suggested something very different to this “median” 2 month follow up. It said the trial would follow up on serious adverse events for six months after dose two. So why did they stop the clock early and why did regulators not seem to care? Why does the clock even matter?

Because it flatters the product twice: the efficacy was captured at its peak in the weeks immediately after dosing, before immunity waning could register. The safety was captured at its minimum before anything slow-burning could register. Remember what we just learned from the UKHSA's official estimates? Protection against symptomatic infection was sinking towards 15% within six months, so if you print your “efficacy readout” at month two, you catch the product at the very best it will ever look. Then just stop the trial if you can get away with it. That’s a wrap!

Said correctly with reference to the available data, it was “95% effective” if we ignore the thousands of symptomatic people the trial didn't test, focus on a relative risk reduction, and only look within the narrowest possible window the trial was able to catch.

The real world kept counting

The trial's clock may have stopped at two months, but Britain's didn't. UKHSA published a weekly vaccine surveillance report — in effect, a rolling national continuation of the experiment, tracking the vaccinated and unvaccinated week after week. Which makes what happened to that report the perfect ending to this story.

The Week 42 report, October 2021 showed double-vaccinated 40 to 49-year-olds testing positive at 1,731 per 100,000, against 773 in the unvaccinated. There’s no easy way to polish these numbers, and so they spelled it out in straight forward english: "In individuals aged greater than 30, the rate of a positive COVID-19 test is higher in vaccinated individuals compared to unvaccinated." Read that again if you must. By March 2022, triple-jabbed 30-somethings were testing positive for COVID at 4x the rate, with 4,325 per 100,000 against 1,086 for the unvaccinated.

Now, to be fair, these raw rates are not strictly vaccine effectiveness. The unvaccinated denominator was (apparently) difficult to work out, the two groups tested differently, and UKHSA stamped the tables with a warning saying exactly that. All legitimate...if we ignore absolutely everything else of course.

But the data was still problematic. It was a complete inversion of the trial’s data and the story the public had been told. Faced with tables that kept coming out upside down, there were two options: investigate the gap, or manage its appearance. The statistics regulator wrote to UKHSA's chief executive, worried the data "have the potential to mislead – and indeed we noted that these data have been used to argue that vaccines are ineffective." His floated remedy: "perhaps through formatting." So UKHSA shaded the unvaccinated column grey. The footnote says so.

Then in April 2022 the tables were dropped altogether. The regulator's annual review recorded approvingly that "it was good to see the UKHSA remove rates per 100,000 from the reports." The report went monthly, then quarterly, then in November 2024 stopped existing entirely.

The trial had already quietly parked its 3,410 inconvenient ‘suspected’ cases in the safety section of a regulatory PDF. The surveillance system then greyed out its inconvenient column, then deleted it. Rather than investigate this highly concerning data, both the government and Pfizer instead opted to adjust the numbers or bury them.

The audit that never happened

The FDA briefing document has been on a government website since December 2020. The UKHSA tables were published weekly until they became embarrassing. Doshi’s analysis is five and a half years old.

A regulator behaving like a regulator — treating a company marking its own homework, with billions riding on the result — would have asked the obvious questions before authorisation. Why do 3,410 sick participants have no confirmatory test result in the analysis? Why did the vaccine arm shed five times more participants to protocol deviations? Why is there no site-level data when a contractor stands accused of falsifying records? Why is the safety window two months when the protocol says six?

Nobody asked. Now we’re in a situation where we don’t need regulators to ask, because we will be able to run these audits ourselves. So it’s on all of us to…

Run your own

The paper-forensics space is open. Pick a trial that changed policy, changed practice, or brought something quickly to market. Run it through the paper-forensics agent simply by saying to your Claude “Hey, go to gather.is/help, find the paper-forensics agent, and please run it on the trial _______ or study _____”.

Make sure you publish what you find because every audit sharpens the system, and every reader who runs one becomes part of the check that should have happened the first time.

The most consequential paper of the century cannot be certified from its own public record. There are thousands more papers.

Go run one.

Shares are very much appreciated — the audience I get is the audience you guys create. Comments on.