Finding errors in high-impact open-access papers
Pipeline design, measured quality-vs-cost frontier, and findings. 2026-10-23 22:18 UTC. Everything below is a model-verified candidate finding; none has yet been confirmed with authors or journals (see "Disclosure").
Summary10 verified papersPipelinePareto frontiers PreregistrationScale runCost at scaleLimitationsDisclosure
Illustrated write-ups of the strongest verified errors (version of record and corrections checked, blind re-derivation, runnable recomputation, figures, independent fact-check).
Downloads: adjudicated_majors.csv (5,726 errors confirmed by two strong models, with quotes and both judges' explanations) · prereg_majors.csv (280 registry discrepancies) · all_experiments.csv (every scored experiment) · README. Model-verified candidates: check against the paper before acting on any row.
Summary
- Cheap models do the detection. On 240 papers with an injected error, GPT-6 Luna (low effort) flags 92% at $0.0022 per paper and Claude Haiku 5.5 (medium) 97% at $0.0055; Opus 5.5, Sonnet 5.5 and GPT-6.1 Sol cost 8–29× as much as Haiku medium and detect about as many (95%–99%). On real errors that a later version of the paper corrected, Luna low ∪ Haiku low (the scale-run screen) finds 66% at $0.0058 per paper and Luna medium ∪ Haiku low 68% at $0.0070.
- One frontier over every experiment. All 44 pipeline experiments that ran on all four benchmarks sit on one chart, scored the same way (share of 460 known major errors their output reports as major). The cost frontier runs from Luna low (generic prompt) (51% at $0.0022 per paper) to Luna med ∪ Haiku med (66% at $0.0090); no expensive single model is on it. Counting precision too, 23 experiments are Pareto-optimal (11 by more than sampling error); the most precise is Luna low ∪ Haiku low → Haiku high → Luna high → Opus high → Sol high (78% of its major flags are known errors, a lower bound, at $0.0501 per paper), but it reports only 23% of the known major errors as major (108/460). Chart and table.
- Confirmed on 381 papers with journal-corrected errors. On a larger benchmark of 381 papers whose errors the journals later corrected (the items that Claude Haiku 5.5 high and GPT-6 Luna high both judged detectable from the original text, a filter that favours those two families), GPT-6 Luna low finds 55% at $0.0022 per paper and Luna-medium ∪ Haiku-high 80% at $0.013; GPT-6.1 Sol and Opus 5.5 reach 70% and 77% at $0.044 and $0.168. End to end, 25% of the major corrected errors survive the cheap verifiers and 20% reach the final Opus + Sol list ($0.039 per benchmark paper). Luna high alone as the cheap verifier keeps 38% (a GPT-6.1 Sol medium verifier 39%, at $0.054 against $0.010 per paper), but after Opus and Sol adjudicate, that variant ends at 22%, so the severity judgement, not detection, is the binding constraint. Without the detectability filter (all 2,448 corrected items in 897 papers, many not visible in the text at all), Luna low ∪ Haiku low finds 30% at any severity and reports 125 of 702 major corrections as major.
- Cross-family agreement filters cheaply; strong judges confirm. Requiring both Haiku 5.5 (high) and GPT-6 Luna (high) to call a flag a real major error cut major flags 9-fold on the scale run (123,861 to 13,497) for $0.0011 per paper. On the four benchmarks the pair raises precision to 72% (a lower bound; 67% with Luna high alone) but keeps fewer known major errors: 128 of 460, against 186 for Luna high alone and 189 for a GPT-6.1 Sol medium verifier at 6× the cost. On a uniform random sample of 120 scale-run candidates, Opus 5.5 (high) and GPT-6.1 Sol (high) both called 43% a real major error (95% CI 35%–52%) and 79% a real error of any severity (71%–85%). The judges differ mainly on severity: Sol alone calls 76% major, Opus alone 50%. Opus + Sol adjudication of the highest-ranked candidates is therefore the step that produces the reportable list. Adversarial check (GPT-6.1 Sol builds the strongest honest defense of the authors, Opus rules) on a random 40 adjudicated majors: 33 (82%, 95% CI 68%–91%) upheld as major, 7 downgraded to minor, 0 judged not an error; on a random 40 from the most recent, lowest-ranked 2,000 adjudications: 30 (75%, 95% CI 60%–86%) upheld as major, 10 downgraded to minor, 0 judged not an error. Downgrades are mostly a slip whose correct value is unambiguous elsewhere in the paper. Independent agent audit (Claude agents with Python and web access; one reproduces the error from scratch, one argues the authors' side, one judges materiality and searches for published corrections) on a random 20: 20/20 reproduced as real errors, 13/20 (65%, 95% CI 43%–82%) also judged major and uncorrected; on the 20 highest-confidence: 20/20 reproduced as real errors, 18/20 (90%, 95% CI 70%–97%) also judged major and uncorrected, 1 already corrected by the journal. Published notices: of 4,084 flagged papers swept (PMC, Crossref, Europe PMC), 273 have a correction, retraction or expression of concern. For 42 confirmed findings the notice fixes the very error we flagged from the original text (both Haiku 5.5 and GPT-6 Luna agree) - independent confirmation by the journals themselves; those rows are marked as already corrected in the downloads. 142 notices concern something else, and 158 are too thin to tell. Author manuscripts: 1,639 of 5,726 confirmed findings (29%) are in papers whose PMC copy is the authors' accepted manuscript rather than the journal's final version. Errors can be fixed during copy-editing: in the write-up checks, 5 of 18 such errors were absent from the publisher's version of record, so roughly 8% of the list may not appear in the final published paper. The downloads flag these rows.
- Expensive models belong at the end. Opus/Sonnet/Sol rate most real errors "minor"; use them as adversarial judges for the top-ranked candidates only, followed by independent checks and a human.
- Measured cost: $0.0042 per paper for screening plus verification on the scale run (Haiku-low screen; $0.0031 + $0.0011), i.e. about $4.2K per million papers on-demand. Bedrock batch (half price) is not offered for these models yet; on-demand throughput quotas, not money, are the binding constraint.
- Deterministic detectors are a complement, not the engine: they caught 4 of 28 known real errors, but registry date/ID checks produced some of the most verifiable findings.
Ten papers with verified major errors
Each passed: cheap full read → Sonnet 5.5 verification with recomputation → Opus 5.5 defense and Opus judge → two fresh independent Opus checks (reproduction from scratch; materiality and existing-correction search) → manual re-derivation of the numbers from the paper text. No correction was found in Crossref, PMC or on the publisher page as of 2026-10-22. Wording is factual and makes no claim about intent.
1. Reported significance of the in-vivo leak result cannot be reproduced from the paper's own counts
The Results state that CSF-leak frequency was significantly higher in all three Fbn2 mutant mouse strains than in wild-type mice ("all comparisons p<0·001"), with relative-risk CIs excluding 1. With the paper's own counts (15/19, 22/29, 8/11 vs 20/35) and the χ² test named in the Methods, p = 0.11, 0.12 and 0.36 (Fisher's exact 0.14–0.49), and the 95% CIs of the relative risks are 0.96–2.00, 0.93–1.89 and 0.80–2.02. The abstract's in-vivo claim of a predisposition to dural rupture rests partly on this comparison.
…frequencies were significantly higher for Fbn2A1052T/+ mice (15 [79%] of 19), RR 1·38, 95% CI 1·14–1·69), for Fbn2D1581V/+ mice (22 [76%] of 29], RR 1·33, 95% CI 1·09–1·63), and for Fbn2M2387T/+ mice (eight [73%] of 11], RR 1·27, 95% CI 1·04–1·58), with all comparisons p<0·001 (figure 4A).
Proportions of mice with leaks were compared between mutant and wild-type mice by use of χ2 tests.
2. Reported "risk ratios" match ratios of mean 1/2 codes, understating the effect roughly tenfold
The binary outcomes were coded 1 (no) and 2 (yes). The reported risk ratios equal the ratio of group means of those codes: at 3 months the reported RR is 0.978, (1+9/561)/(1+20/491) = 0.976, whereas the risk ratio from the same counts is 0.39. The pattern holds at every time point (reported 0.96–0.98 vs 0.39–0.67 from counts), so the Discussion's "about a 3% reduction in risk" understates relative reductions of roughly 33–61%.
…responses coded 1 for respondents not reporting ever experiencing the event and 2 for those reporting ever experiencing the event.
at 3 months (9 of 561 [2%] vs 20 of 491 [4%]; risk ratio [RR], 0.978; 95% CI, 0.959-0.997)
The magnitude of the intervention effects in this trial was modest, about a 3% reduction in risk.
3. A miscomputed registry proportion reverses a reported under-representation
The overall registry (CiNA) share of Asian or Pacific Islander patients is given as 5.9%, but the paper's own counts give 145,568/4,891,486 = 2.98%, consistent with its period-specific figures (2.7% and 3.4%). With either trial figure the paper reports (272/8,308 = 3.3% or 371/8,309 = 4.5%), Asian or Pacific Islander participants were over-, not under-represented overall, so the reported difference of −1.5 percentage points (P < .001) does not follow.
…an underrepresentation of Asian or Pacific Islander participants (trials, 272 of 8308 [4.5%]; CiNA, 145 568 of 4 891 486 [5.9%]; difference, -1.5 percentage points; P < .001)
4. An impossible confidence interval behind a "significant" stroke reduction
The Abstract and Results give the 9 µg/m³ stroke risk difference as −2.26 per mille (95% CI −8.97 to −20.64), an interval that excludes its own estimate. Table 2 gives −8.97 to +20.64, which includes zero, so the Key Points statement that the intervention would significantly reduce stroke hospitalization at this threshold is not supported by the paper's table. Two further CI sign discrepancies appear between abstract, text and table.
…the estimated 5-year risk difference of hospitalization for stroke was -2.26 per mille (95% CI, -8.97 to -20.64 per mille)
Table 2: ≤9 | 10.06 (3.37 to 32.97) | -2.26 (-8.97 to 20.64)
5. Implausibly narrow confidence intervals underpin associations reported in the abstract
With 76 catastrophic-expenditure cases among 296 participants, the abstract reports adjusted ORs of 2.0 (95% CI 1.9–2.1) for the second poverty quintile and 9.7 (9.6–9.8) for subcounty hospitals. The table counts imply standard errors 15–75 times larger (crude 95% CIs 0.52–2.81 and 4.1–19.5), and the unadjusted second-quintile OR is 1.4 (0.8–2.4), p = 0.30. The abstract also disagrees with Table 2 (9.6–9.8 vs 9.7–9.9).
…belonging to the second poverty quintile (2·0, 1·9–2·1, p<0·0001), and seeking care from subcounty hospitals (9·7, 9·6–9·8, p<0·0001)…
Table 2: Second quintile | 1·4 (0·8–2·4) | 0·30 | 2·0 (1·9–2·1) | <0·0001
6. The abstract's age pattern is contradicted by the paper's own syphilis estimates
The abstract states that, across the three primary region/population groups, prevalence was higher in 15–24-year-old than 25–49-year-old women for all STIs except HSV-2, and the author summary adds "regardless of region or population type". The Results report overall syphilis prevalence among South African clinic/community women of 4.3% (95% CI 2.5–7.2) at ages 25–49 versus 1.5% (1.0–2.2) at 15–24, with non-overlapping intervals, and a higher point estimate for older women in Southern/Eastern Africa as well (2.0% vs 1.4%).
Across the three primary region/population groups (South Africa community-based, Southern/Eastern Africa community-based, and Eastern Africa higher-risk), prevalence was higher among 15–24-year-old than 25–49-year-old women for all STIs except HSV-2.
…the summary estimate of syphilis prevalence overall among 25–49-year-old women (4.3% [95% CI: 2.5%, 7.2%]; I2 = 94.8%) was higher than that among younger women (1.5%) (Fig 6).
7. Primary-outcome denominators exceed the population the outcome is defined for
The hyperglycemia primary outcome is defined for patients with diabetes ("patients without diabetes were excluded from analysis"), but its denominators are 8,676 (intervention) and 8,815 (control), while Table 1 lists 2,824 and 2,909 patients with diabetes (at most 5,713 and 5,887 even if every patient with missing diabetes status had diabetes). The denominators equal all cases with glucose data in the flowchart, so the reported primary outcome appears to include patients without diabetes.
For glycemic outcomes, patients without diabetes were excluded from analysis.
Table 2: No postoperative hyperglycemia | 7445/8676 (85.8) | 7559/8815 (85.8)
Table 1: Diabetes | 2909/10 296 (28.3) | 2824/10 091 (28.0)
8. The abstract reports phantom-sensation scores as range of motion in degrees
The abstract states that one surgical cohort reported "significantly greater phantom range of motion" (12.47 ± 2.41 vs 10.14 ± 1.45 degrees). The Results identify 12.47 ± 2.41 vs 10.14 ± 1.9 as unitless phantom-sensation scores (graded 1–3 per foot region), while phantom range of motion is reported as a percentage of the contralateral limb. The abstract therefore attaches the wrong measure and unit (and a different SD) to a headline comparison.
AB subjects reported significantly greater phantom range of motion postamputation (AB: 12.47 ± 2.41, TB: 10.14 ± 1.45 degrees) when compared to TB subjects (P < 0.05).
AB subjects reported significantly higher phantom sensations (12.47 ± 2.41, P = 0.03) compared to TB subjects (10.14 ± 1.9) (P < 0.05, Student's t test).
9. The abstract claims a blood-pressure benefit that the trial's own results do not show
The abstract says intergroup differences favouring the intervention were noted for diastolic blood pressure. Table 2 gives diastolic differences of −2.6 mm Hg (95% CI −6.2 to 0.9; P = .15) at 3 months and −2.4 (−5.7 to 1.0; P = .17) at 6 months, and the Results state that between-group differences in blood pressure were not observed at 6 months.
Intergroup differences favoring the intervention were also noted for fasting blood glucose, diastolic blood pressure, and dietary changes.
Between-group differences in blood pressure and lipids were not observed at 6 months.
Table 2: 6 mo | 139 | -3.1 (9.2) | -5.4 (11.1) | -2.4 (-5.7 to 1.0) | .17
10. Effect modification is declared significant against the paper's own stated criterion
The Methods define subgroup estimates as significantly different only if their 95% CIs do not overlap. The Results nevertheless declare the food-insecure (−13.6; 95% CI −25.0 to −2.3) and food-secure (5.0; −15.8 to 25.7) estimates statistically significantly different, although the intervals overlap from −15.8 to −2.3; a Wald test on the reported intervals gives p ≈ 0.12. The conclusion that food insecurity modifies the association rests on this comparison.
PATEs were considered statistically significantly different if their respective 95% CIs did not overlap.
The observed PATE estimates among the food-secure vs food-insecure subgroups were statistically significantly different, suggesting that food insecurity modified the association between SNAP and nonadherence to antihypertensive medication therapy.
Pipeline
Recommended configuration for all open-access high-quality papers: detectors on everything; screen with GPT-6 Luna medium ∪ Claude Haiku 5.5 low (64% of known major errors at $0.0067 per paper), or Luna low alone when cost-limited (46% at $0.0022); verify with Haiku 5.5 high → GPT-6 Luna high; rank candidates with the learned ranker (verifier confidences, category, text); Opus 5.5 + GPT-6.1 Sol adjudicate from the top down (both must call it major); sample-check with a Sol defence and an Opus judge; independent hand-checks (version of record, blind re-derivation) and human review before contact. Registration track for every trial with an own registry ID.
Code: src/llm/bedrock.py (client, ledger, AIMD rate gates), src/pipeline/ (stages, experiment runner, scale, cascade daemon, prereg), src/detectors/, src/eval/ (seeded and real-correction benchmarks, scorer), workflows/ (agentic tiers used for the first 10 papers).
Quality vs cost over all experiments
Every pipeline experiment run on the benchmarks is scored the same way, as an end-to-end pipeline, with one quality measure: of the 460 known major errors in the four benchmarks (Seeded errors, PMC-version corrections, Verified real errors, Journal-corrected errors), the share that the pipeline's output reports as a major error. The 44 experiments that ran on all four benchmarks are ranked on one frontier: 18 single screening models, 5 two-model unions, 12 screen + verifier combinations (9 verifiers after the scale-run screen Luna low ∪ Haiku low: Haiku high, Haiku high (materiality rubric), Haiku med, Luna 5.6 high, Luna high, Luna high (materiality rubric), Luna med, Sol med, gpt-oss; and Luna med → Haiku high, Haiku med → Haiku high, Luna med ∪ Haiku low → Haiku high) and 9 multi-stage pipelines, including the production chain through Opus 5.5 and GPT-6.1 Sol. A flag that both screeners of a union raised is verified once, as in the scale run. Verifiers compared only flag by flag (Opus med, Sonnet med, Terra med) are in the table at the end of this section. Cost is the LLM spend of all stages, averaged per benchmark paper, on one price table. All 181 benchmark × experiment results are in all_experiments.csv, published with the report.
Cost frontier for recall (★): 1 Luna low (generic prompt) ($0.0022, 50.9%, 234/460) · 2 Haiku low (generic prompt) ($0.0033, 54.8%, 252/460) · 3 Haiku low (reconcile prompt) ($0.0044, 58.9%, 271/460) · 4 Luna low ∪ Haiku low ($0.0056, 63.5%, 292/460) · 5 Luna med ∪ Haiku low (generic prompt) ($0.0067, 65.4%, 301/460) · 6 Luna med ∪ Haiku med ($0.0090, 66.3%, 305/460).
On the 225 real major errors alone (seeded errors left out), the cost frontier is close to it: it keeps 5 of its 6 points, drops Haiku low (generic prompt) and adds Luna high, Luna med ∪ Haiku high. Haiku low was run twice with the same settings (250 and 252 of 460 known major errors, $0.0034 and $0.0034 per paper): differences of that size are run-to-run noise, so the repeat run is not ranked.
Recall alone favours raw screeners, which report many unconfirmed flags. The second chart plots precision: the share of output major flags that match a known error (a lower bound, because these papers hold other, unknown errors too). Experiments ringed in both charts are on the three-way frontier: no other experiment is at least as cheap, finds at least as many known major errors and is at least as precise, and strictly better on one of these (23 experiments, ◆ in the table). Many of them win by margins inside sampling error; allowing 2% on recall and precision, 11 remain: Luna low (generic prompt), Luna low, Haiku low (generic prompt), Haiku low, Haiku low (reconcile prompt), Luna med → Haiku high, Luna low ∪ Haiku low, Luna med → Haiku high → Luna high, Luna low ∪ Haiku low → Haiku med, Luna low ∪ Haiku low → Luna med, Luna med ∪ Haiku med.
Cost frontier for precision: 1 Luna low (generic prompt) ($0.0022, 59.2%, 263/444) · 2 Luna low ($0.0022, 63.6%, 246/387) · 3 Luna med → Haiku high ($0.0049, 70.2%, 127/181) · 4 Luna med → Haiku high → Luna high ($0.0057, 74.4%, 119/160) · 5 Luna low ∪ Haiku low → Haiku high → Luna high → Opus high ($0.0382, 76.2%, 141/185) · 6 Luna low ∪ Haiku low → Haiku high → Luna high → Opus high → Sol high ($0.0501, 77.7%, 139/179).
Table: every ranked experiment (44)
| Experiment (shape = kind) | $/paper | Known major errors reported as major [95% CI] (of 460) | …real errors only | All known errors, any severity (of 1120) | Precision (lower bound) | Major flags per clean paper | Failed screen rows | Benchmarks | |
|---|---|---|---|---|---|---|---|---|---|
| ★◆ | Luna low (generic prompt) | $0.0022 | 51% [46%–55%] (234/460) | 44% | 62% | 59% | 0.35 | 1 | all 4 |
| ◆ | Luna low | $0.0022 | 46% [42%–51%] (213/460) | 37% | 62% | 64% | 0.28 | 1 | all 4 |
| Luna low (reconcile prompt) | $0.0028 | 50% [45%–55%] (230/460) | 43% | 60% | 59% | 0.38 | 1 | all 4 | |
| ★◆ | Haiku low (generic prompt) | $0.0033 | 55% [50%–59%] (252/460) | 40% | 57% | 37% | 0.88 | all 4 | |
| Luna med | $0.0033 | 49% [44%–53%] (225/460) | 43% | 67% | 57% | 0.35 | all 4 | ||
| ◆ | Haiku low (detector hints) | $0.0034 | 51% [47%–56%] (235/460) | 39% | 59% | 46% | 0.57 | all 4 | |
| ◆ | Haiku low | $0.0034 | 54% [50%–59%] (250/460) | 40% | 59% | 44% | 0.65 | all 4 | |
| gpt-oss | $0.0034 | 52% [48%–57%] (240/460) | 37% | 41% | 42% | 0.72 | all 4 | ||
| ★◆ | Haiku low (reconcile prompt) | $0.0044 | 59% [54%–63%] (271/460) | 47% | 63% | 45% | 0.66 | all 4 | |
| ◆ | Luna med → Haiku high | $0.0049 | 29% [25%–33%] (132/460) | 25% | 26% | 70% | 0.06 | all 4 | |
| ◆ | Luna high | $0.0052 | 52% [47%–56%] (239/460) | 48% | 72% | 55% | 0.42 | 1 | all 4 |
| ★◆ | Luna low ∪ Haiku low | $0.0056 | 63% [59%–68%] (292/460) | 53% | 73% | 51% | 0.93 | 1 | all 4 |
| Haiku med | $0.0056 | 59% [55%–64%] (273/460) | 48% | 70% | 48% | 0.59 | 16 | all 4 | |
| Luna 5.6 low | $0.0056 | 56% [52%–61%] (259/460) | 44% | 65% | 49% | 0.57 | 2 | all 4 | |
| ◆ | Luna med → Haiku high → Luna high | $0.0057 | 27% [23%–31%] (125/460) | 24% | 13% | 74% | 0.06 | all 4 | |
| ★◆ | Luna med ∪ Haiku low (generic prompt) | $0.0067 | 65% [61%–70%] (301/460) | 56% | 76% | 44% | 1.23 | all 4 | |
| ◆ | Luna med ∪ Haiku low | $0.0067 | 64% [60%–68%] (295/460) | 55% | 76% | 49% | 1.00 | all 4 | |
| Haiku med (reconcile prompt) | $0.0072 | 60% [55%–64%] (275/460) | 52% | 72% | 44% | 0.66 | 90 | all 4 | |
| Haiku med → Haiku high | $0.0078 | 29% [25%–33%] (134/460) | 26% | 31% | 62% | 0.11 | 16 | all 4 | |
| Luna low ∪ Haiku low → gpt-oss | $0.0083 | 19% [16%–23%] (89/460) | 15% | 37% | 72% | 0.05 | 1 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Haiku med | $0.0085 | 32% [28%–37%] (148/460) | 29% | 37% | 68% | 0.12 | 1 | all 4 |
| ◆ | Luna low ∪ Haiku low → Luna med | $0.0088 | 40% [36%–45%] (186/460) | 38% | 35% | 68% | 0.21 | 1 | all 4 |
| Haiku med → Haiku high → Luna high | $0.0088 | 25% [21%–29%] (114/460) | 23% | 13% | 66% | 0.06 | 16 | all 4 | |
| Luna low ∪ Haiku low → Haiku high (materiality rubric) | $0.0089 | 15% [12%–18%] (68/460) | 13% | 37% | 58% | 0.05 | 1 | all 4 | |
| Luna low ∪ Haiku low → Haiku high | $0.0089 | 31% [27%–35%] (143/460) | 27% | 36% | 68% | 0.15 | 1 | all 4 | |
| ★◆ | Luna med ∪ Haiku med | $0.0090 | 66% [62%–70%] (305/460) | 58% | 81% | 51% | 0.94 | 16 | all 4 |
| Luna low ∪ Haiku low → Luna high (materiality rubric) | $0.0101 | 20% [16%–24%] (91/460) | 20% | 37% | 70% | 0.03 | 1 | all 4 | |
| Luna low ∪ Haiku low → Luna high | $0.0101 | 40% [36%–45%] (186/460) | 36% | 36% | 67% | 0.22 | 1 | all 4 | |
| Haiku high | $0.0102 | 60% [55%–64%] (274/460) | 52% | 77% | 45% | 0.62 | 31 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high | $0.0103 | 28% [24%–32%] (128/460) | 24% | 14% | 72% | 0.10 | 1 | all 4 |
| Luna med ∪ Haiku low → Haiku high | $0.0103 | 33% [29%–37%] (152/460) | 31% | 36% | 66% | 0.15 | all 4 | ||
| ◆ | Luna med ∪ Haiku low → Haiku high → Luna high | $0.0117 | 30% [26%–35%] (140/460) | 26% | 15% | 72% | 0.11 | all 4 | |
| Luna low ∪ Haiku low → Haiku high → Luna high (re-grades every flag called real) | $0.0130 | 40% [35%–44%] (183/460) | 35% | 35% | 68% | 0.21 | 1 | all 4 | |
| Luna med ∪ Haiku high | $0.0136 | 66% [61%–70%] (302/460) | 60% | 84% | 50% | 0.97 | 31 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Luna 5.6 high | $0.0190 | 41% [36%–45%] (188/460) | 37% | 36% | 64% | 0.24 | 1 | all 4 |
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high → Opus high | $0.0382 | 24% [20%–28%] (110/460) | 20% | 13% | 76% | 0.04 | 1 | all 4 |
| Sol low | $0.0454 | 52% [47%–56%] (237/460) | 56% | 75% | 45% | – | all 4 | ||
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high → Opus high → Sol high | $0.0501 | 23% [20%–28%] (108/460) | 20% | 11% | 78% | 0.04 | 1 | all 4 |
| Sonnet low | $0.0518 | 37% [33%–42%] (172/460) | 31% | 64% | 63% | – | all 4 | ||
| Sonnet med | $0.0553 | 38% [34%–42%] (174/460) | 34% | 67% | 61% | – | all 4 | ||
| ◆ | Luna low ∪ Haiku low → Sol med | $0.0600 | 41% [37%–46%] (189/460) | 36% | 36% | 67% | 0.20 | 1 | all 4 |
| Luna low ∪ Haiku low → Luna high → Opus high | $0.0606 | 27% [23%–31%] (122/460) | 22% | 20% | 74% | 0.06 | 1 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Luna high → Opus high → Sol high | $0.0750 | 26% [22%–30%] (118/460) | 21% | 12% | 75% | 0.05 | 1 | all 4 |
| ◆ | Opus med | $0.1727 | 46% [42%–51%] (212/460) | 42% | 81% | 65% | – | 2 | all 4 |
★ cost frontier for major-error recall · ◆ three-way frontier. Pooled Wilson intervals treat errors as independent. Major flags per clean paper is measured on the 240 unmodified control papers of the seeded benchmark (flags there include real errors in the originals, so it is an upper bound on false alarms; – where controls were not run: Opus, Sol and Sonnet screened the 240 papers with an injected error only). Papers whose screener output could not be used count as finding nothing, and unions and cascades built on that screener inherit them (column 'Failed screen rows'): Haiku med (reconcile prompt) 90 of 1002, Haiku high 31 of 1002, Haiku med 16 of 1002, Luna 5.6 low 2 of 1002, Opus med 2 of 762, Luna low (generic prompt) 1 of 1002. Most were cut off at the 16,000-token output cap; GPT-6 Luna rows that failed only because of throttling were re-run, and Opus 5.5 refusals were re-asked of Opus 5.
Not ranked: repeat runs and experiments not run on all four benchmarks (2)
| Experiment (shape = kind) | $/paper | Known major errors reported as major [95% CI] (of 460) | …real errors only | All known errors, any severity (of 1120) | Precision (lower bound) | Major flags per clean paper | Failed screen rows | Benchmarks | |
|---|---|---|---|---|---|---|---|---|---|
| Haiku low (repeat run) | $0.0034 | 55% [50%–59%] (252/460) | 43% | 61% | 44% | 0.64 | 1 | all 4 | |
| Luna med ∪ Haiku low → Sol med | $0.1408 | 74% [51%–88%] (14/19) | 74% | 86% | 65% | – | Verified real errors |
Pooled over the benchmarks each ran on; rows on fewer benchmarks are not comparable with the ranked rows.
Verifiers judged flag by flag
A second view of the verifiers above (except Luna med, not run on this flag pool): agreement with the Opus 5.5 + GPT-6.1 Sol oracle on the found-set flag pool; ★ marks the cost frontier for agreement on real vs not (acc) and for recall of oracle-major flags. The two Oracle rows are the oracle runs themselves (100% agreement by construction in the first three columns; the last column is scored against the known errors).
| Verifier | $/flag | Agrees with oracle (real?) | Precision on major | Recall on major | Keeps known majors major |
|---|---|---|---|---|---|
| ★ Haiku high (materiality rubric) | $0.0029 | 88% ★ | 95% | 44% ★ | 50% |
| ★ Haiku med | $0.0030 | 86% | 96% | 61% ★ | 58% |
| gpt-oss | $0.0033 | 85% | 86% | 44% | 46% |
| ★ Haiku high | $0.0034 | 88% | 97% | 71% ★ | 69% |
| ★ Luna high (materiality rubric) | $0.0040 | 94% ★ | 100% | 49% | 54% |
| ★ Luna high | $0.0045 | 95% ★ | 93% | 93% ★ | 85% |
| ★ Luna 5.6 high | $0.0119 | 94% | 85% | 95% ★ | 88% |
| ★ Terra med | $0.0437 | 95% ★ | 88% | 85% | 77% |
| Sonnet med | $0.0501 | 90% | 100% | 17% | 15% |
| ★ Sol med | $0.0549 | 99% ★ | 95% | 98% ★ | 88% |
| Oracle: Sol high | $0.0688 | 100% | 100% | 100% | 88% |
| Opus med | $0.1068 | 98% | 100% | 90% | 85% |
| Oracle: Opus high | $0.1138 | 100% | 100% | 100% | 73% |
Not on this frontier, because they ran on scale-run candidates, which have no ground truth: the order in which candidates are adjudicated (learned ranker vs baseline, A/B), the adversarial defence of adjudicated majors (GPT-6.1 Sol defends, Opus 5.5 rules), the Opus 5 fallback for Opus 5.5 refusals, and the materiality rubric at scale (its benchmark versions, the materiality-rubric verifiers, are ranked above). Per-benchmark screener frontiers and the stage-by-stage view on journal-corrected errors follow below.
Screeners, benchmark by benchmark
Screeners on seeded errors (240 mutated + 240 control papers)
Frontier: 1 Luna low (generic) ($0.0021, 91%) · 2 Luna low ($0.0022, 92%) · 3 Luna med ($0.0033, 95%) · 4 Luna high ($0.0052, 98%) · 5 Luna med ∪ Haiku low (generic) ($0.0067, 99%) · 6 Luna med ∪ Haiku med ($0.0091, 100%)
Recall at any severity; LLM-assisted matching; 95% bootstrap CIs in the table view.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0021 | 91% [88–95] | 57% | 0.35 |
| ★ luna6-low-checklist | OpenAI | $0.0022 | 92% [88–95] | 54% | 0.28 |
| luna6-low-reconcile | OpenAI | $0.0028 | 92% [88–95] | 56% | 0.38 |
| ★ luna6-med-checklist | OpenAI | $0.0033 | 95% [92–98] | 53% | 0.35 |
| haiku55-low-generic | Claude | $0.0034 | 90% [86–94] | 68% | 0.88 |
| haiku55-low-hints | Claude | $0.0034 | 93% [89–96] | 62% | 0.57 |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 88% [84–92] | 65% | 0.64 |
| haiku55-low-checklist | Claude | $0.0035 | 88% [84–92] | 67% | 0.65 |
| gptoss120-checklist | OpenAI (open-weight) | $0.0035 | 75% [69–80] | 66% | 0.72 |
| haiku55-low-reconcile | Claude | $0.0046 | 92% [89–96] | 69% | 0.66 |
| ★ luna6-high-checklist | OpenAI | $0.0052 | 98% [96–100] | 54% | 0.42 |
| luna56-low-checklist | OpenAI | $0.0056 | 90% [86–94] | 66% | 0.57 |
| luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0057 | 96% [93–98] | 72% | 0.93 |
| haiku55-med-checklist | Claude | $0.0058 | 97% [95–99] | 69% | 0.59 |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0067 | 99% [98–100] | 73% | 1.23 |
| luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0068 | 97% [95–99] | 71% | 1.00 |
| haiku55-med-reconcile | Claude | $0.0075 | 90% [87–94] | 65% | 0.66 |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0091 | 100% [100–100] | 73% | 0.94 |
| haiku55-high-checklist | Claude | $0.0106 | 97% [95–99] | 65% | 0.62 |
| luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0139 | 100% [99–100] | 70% | 0.97 |
| sol61-low-checklist | OpenAI | $0.0457 | 98% [97–100] | 46% | |
| sonnet55-low-checklist | Claude | $0.0510 | 96% [93–98] | 42% | |
| sonnet55-med-checklist | Claude | $0.0552 | 95% [92–98] | 40% | |
| opus55-med-checklist | Claude | $0.1590 | 99% [97–100] | 50% |
Screeners on real corrections detectable from the original text (113 papers, 205 values)
Frontier: 1 Luna low (generic) ($0.0023, 54%) · 2 Luna med ($0.0036, 57%) · 3 Luna high ($0.0056, 60%) · 4 Haiku med ($0.0058, 62%) · 5 Luna low ∪ Haiku low ($0.0058, 66%) · 6 Luna med ∪ Haiku low ($0.0070, 68%) · 7 Luna med ∪ Haiku low (generic) ($0.0070, 68%) · 8 Luna med ∪ Haiku med ($0.0093, 73%) · 9 Luna med ∪ Haiku high ($0.0142, 75%)
Values a later version corrected; most are minor, so recall is reported at any severity.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0023 | 54% [43–63] | 10% | |
| luna6-low-checklist | OpenAI | $0.0024 | 52% [41–61] | 9% | |
| luna6-low-reconcile | OpenAI | $0.0028 | 45% [34–55] | 11% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0034 | 36% [23–48] | 20% | |
| haiku55-low-checklist | Claude | $0.0034 | 48% [37–58] | 5% | |
| haiku55-low-hints | Claude | $0.0034 | 49% [38–59] | 4% | |
| haiku55-low-generic | Claude | $0.0035 | 51% [42–58] | 11% | |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 52% [41–61] | 5% | |
| ★ luna6-med-checklist | OpenAI | $0.0036 | 57% [47–66] | 11% | |
| haiku55-low-reconcile | Claude | $0.0045 | 47% [38–56] | 11% | |
| ★ luna6-high-checklist | OpenAI | $0.0056 | 60% [51–69] | 6% | |
| ★ haiku55-med-checklist | Claude | $0.0058 | 62% [53–71] | 8% | |
| ★ luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0058 | 66% [56–74] | 13% | |
| luna56-low-checklist | OpenAI | $0.0060 | 60% [49–68] | 20% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0070 | 68% [59–75] | 13% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0070 | 68% [59–75] | 17% | |
| haiku55-med-reconcile | Claude | $0.0075 | 65% [56–74] | 7% | |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0093 | 73% [64–81] | 15% | |
| haiku55-high-checklist | Claude | $0.0106 | 68% [56–77] | 8% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0142 | 75% [66–83] | 15% | |
| sol61-low-checklist | OpenAI | $0.0477 | 62% [53–71] | 15% | |
| sonnet55-low-checklist | Claude | $0.0573 | 52% [43–61] | 3% | |
| sonnet55-med-checklist | Claude | $0.0593 | 60% [51–69] | 1% | |
| opus55-med-checklist | Claude | $0.2165 | 73% [63–81] | 10% |
Screeners on journal-corrected errors detectable from the original text (381 papers, 647 values)
Frontier: 1 Luna low (generic) ($0.0021, 54%) · 2 Luna low ($0.0022, 55%) · 3 Luna med ($0.0033, 60%) · 4 Luna high ($0.0050, 67%) · 5 Luna med ∪ Haiku low (generic) ($0.0065, 70%) · 6 Luna med ∪ Haiku low ($0.0065, 70%) · 7 Luna med ∪ Haiku med ($0.0086, 75%) · 8 Luna med ∪ Haiku high ($0.0129, 80%)
Errors a journal corrected in a published notice and that two models judge detectable from the original; recall at any severity.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0021 | 54% [49–59] | 23% | |
| ★ luna6-low-checklist | OpenAI | $0.0022 | 55% [49–60] | 20% | |
| luna6-low-reconcile | OpenAI | $0.0027 | 52% [47–57] | 19% | |
| haiku55-low-generic | Claude | $0.0032 | 46% [41–52] | 22% | |
| haiku55-low-checklist | Claude | $0.0032 | 50% [45–56] | 23% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0032 | 30% [25–35] | 22% | |
| haiku55-low-checklist-s1 | Claude | $0.0032 | 53% [48–58] | 24% | |
| haiku55-low-hints | Claude | $0.0032 | 49% [44–54] | 20% | |
| ★ luna6-med-checklist | OpenAI | $0.0033 | 60% [56–65] | 19% | |
| haiku55-low-reconcile | Claude | $0.0041 | 57% [52–62] | 26% | |
| ★ luna6-high-checklist | OpenAI | $0.0050 | 67% [62–71] | 26% | |
| haiku55-med-checklist | Claude | $0.0053 | 62% [58–66] | 23% | |
| luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0054 | 67% [62–71] | 32% | |
| luna56-low-checklist | OpenAI | $0.0055 | 57% [52–63] | 25% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0065 | 70% [66–74] | 30% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0065 | 70% [66–75] | 30% | |
| haiku55-med-reconcile | Claude | $0.0067 | 66% [61–72] | 28% | |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0086 | 75% [72–79] | 30% | |
| haiku55-high-checklist | Claude | $0.0096 | 72% [67–76] | 24% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0129 | 80% [77–84] | 30% | |
| sol61-low-checklist | OpenAI | $0.0442 | 70% [66–75] | 29% | |
| sonnet55-low-checklist | Claude | $0.0506 | 55% [51–60] | 14% | |
| sonnet55-med-checklist | Claude | $0.0540 | 57% [53–62] | 14% | |
| opus55-med-checklist | Claude | $0.1684 | 77% [73–81] | 17% |
Screeners on independently verified real errors (28 papers: 19 major, 9 minor)
Frontier: 1 Luna low (generic) ($0.0023, 61%) · 2 Luna low (reconcile) ($0.0031, 64%) · 3 Haiku low (generic) ($0.0034, 86%) · 4 Luna low ∪ Haiku low ($0.0059, 89%) · 5 Luna med ∪ Haiku low (generic) ($0.0071, 93%) · 6 Luna med ∪ Haiku low ($0.0073, 96%) · 7 Luna med ∪ Haiku high ($0.0149, 100%)
Recall at any severity; LLM-assisted matching. The set was first found by Haiku-based reading, so it favours Haiku; it has no clean controls.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0023 | 61% [43–79] | 57% | |
| luna6-low-checklist | OpenAI | $0.0023 | 61% [43–79] | 57% | |
| ★ luna6-low-reconcile | OpenAI | $0.0031 | 64% [46–82] | 54% | |
| ★ haiku55-low-generic | Claude | $0.0034 | 86% [71–96] | 82% | |
| haiku55-low-hints | Claude | $0.0034 | 79% [61–93] | 75% | |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 79% [64–93] | 68% | |
| haiku55-low-checklist | Claude | $0.0036 | 79% [61–93] | 75% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0036 | 43% [25–61] | 32% | |
| luna6-med-checklist | OpenAI | $0.0037 | 79% [64–93] | 75% | |
| haiku55-low-reconcile | Claude | $0.0049 | 86% [73–96] | 79% | |
| ★ luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0059 | 89% [79–100] | 86% | |
| luna56-low-checklist | OpenAI | $0.0060 | 57% [39–75] | 57% | |
| haiku55-med-checklist | Claude | $0.0061 | 86% [71–96] | 79% | |
| luna6-high-checklist | OpenAI | $0.0062 | 75% [61–89] | 75% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0071 | 93% [82–100] | 93% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0073 | 96% [89–100] | 93% | |
| haiku55-med-reconcile | Claude | $0.0080 | 86% [71–96] | 82% | |
| luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0098 | 96% [89–100] | 89% | |
| haiku55-high-checklist | Claude | $0.0113 | 93% [82–100] | 86% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0149 | 100% [100–100] | 96% | |
| sol61-low-checklist | OpenAI | $0.0496 | 89% [75–100] | 89% | |
| sonnet55-low-checklist | Claude | $0.0530 | 79% [64–93] | 39% | |
| sonnet55-med-checklist | Claude | $0.0590 | 82% [68–96] | 57% | |
| opus55-med-checklist | Claude | $0.1738 | 93% [82–100] | 68% |
End-to-end pipeline frontier on journal-corrected errors
| Stage | Cumulative $/paper | Major corrected errors still held as major (95% CI) | All corrected errors | Flags/paper passed on | Precision of this stage's output (measured on the scale run) |
|---|---|---|---|---|---|
| Screen: Luna-low ∪ Haiku-low (any flag) | $0.0054 | 74% [66%–81%] (126/170) | 67% | 6.96 | |
| Screen: flagged major | $0.0054 | 57% [48%–66%] (97/170) | 32% | 1.16 | |
| + Haiku 5.5 high verifies | $0.0084 | 29% [19%–38%] (49/170) | 11% | 0.28 | |
| + GPT-6 Luna high verifies (candidates) | $0.0095 | 25% [16%–35%] (42/170) | 9% | 0.21 | 43% both-major, 79% real (random scale sample, Opus+Sol) |
| + Opus 5.5 high adjudicates | $0.0306 | 21% [12%–31%] (35/170) | 7% | 0.15 | |
| + GPT-6.1 Sol high adjudicates (final list) | $0.0392 | 20% [11%–30%] (34/170) | 7% | 0.14 | 20/20 real, 13/20 major and uncorrected (agent audit); 33/40 upheld under adversarial review |
| Variant: Haiku-high only removes non-errors; Luna-high makes the severity call | $0.0122 | 36% [28%–47%] (62/170) | 16% | 0.41 | |
| Variant: Luna-high is the only cheap verifier | $0.0096 | 38% [29%–48%] (64/170) | 16% | 0.42 | |
| Variant: Luna-high only, then Opus + Sol adjudicate (final list) | $0.0667 | 22% [13%–32%] (37/170) | 8% | 0.18 | |
| Alternative: GPT-6.1 Sol medium is the only verifier | $0.0536 | 39% [29%–50%] (67/170) | 15% | 0.44 |
381 papers whose errors the journals later corrected, restricted to errors a careful reader could have spotted in the original (both Claude Haiku 5.5 and GPT-6 Luna agree). Each verification stage passes on only the flags it rates major, so recall can only fall while precision rises; costs are per benchmark paper, which carry more flags than an average paper. The first row counts a corrected error as found if any screener flag points at it.
Verifiers
Pool: 244 screener flags. Oracle labels: 184 real (41 major), 36 not real, 24 oracle disagreements.
| Verifier | $/flag | Agrees with oracle (real?) | Precision on major | Recall on major | Confirms known errors | Keeps known majors major |
|---|---|---|---|---|---|---|
| haiku55-high-material | $0.0029 | 88% | 95% | 44% | 95% | 50% |
| haiku55-med | $0.0030 | 86% | 96% | 61% | 95% | 58% |
| gptoss120 | $0.0033 | 85% | 86% | 44% | 98% | 46% |
| haiku55-high | $0.0034 | 88% | 97% | 71% | 98% | 69% |
| luna6-high-material | $0.0040 | 94% | 100% | 49% | 93% | 54% |
| luna6-high | $0.0045 | 95% | 93% | 93% | 93% | 85% |
| luna56-high | $0.0119 | 94% | 85% | 95% | 93% | 88% |
| terra56-med | $0.0437 | 95% | 88% | 85% | 90% | 77% |
| sonnet55-med | $0.0501 | 90% | 100% | 17% | 98% | 15% |
| sol61-med | $0.0549 | 99% | 95% | 98% | 95% | 88% |
| oracle-sol61-high | $0.0688 | 100% | 100% | 100% | 95% | 88% |
| opus55-med | $0.1068 | 98% | 100% | 90% | 93% | 85% |
| oracle-opus55-high | $0.1138 | 100% | 100% | 100% | 93% | 73% |
Oracle = agreement of Opus 5.5 (high) and GPT-6.1 Sol (high); the Sol-medium row is correlated with half of the oracle.
Severity-rubric experiment. The default rubric calls any error in the abstract or a headline result major; a 'materiality' rubric (major only if correcting it changes a conclusion, significance or a headline estimate materially) was tried on the cheap verifiers to see whether they then agree better with Opus+Sol. It does not: precision against Opus+Sol barely moves while half the confirmed majors are dropped, and on the found set it keeps far fewer known major errors (rows ending in -material above). The default rubric stays; strong-model adjudication of top-ranked candidates does the severity call.
| Extra filter on two-model candidates | Kept | Opus+Sol both major | Share of Opus+Sol majors kept |
|---|---|---|---|
| No extra filter (random candidates) | 120/120 | 43% | 100% |
| haiku55-high-material (random candidates) | 53/120 | 53% | 54% |
| luna6-high-material (random candidates) | 58/120 | 57% | 63% |
| haiku55-high-material & luna6-high-material (random candidates) | 36/120 | 61% | 42% |
| No extra filter (top-ranked candidates) | 200/200 | 68% | 100% |
| haiku55-high-material (top-ranked candidates) | 102/200 | 75% | 56% |
| luna6-high-material (top-ranked candidates) | 98/200 | 68% | 49% |
| haiku55-high-material & luna6-high-material (top-ranked candidates) | 68/200 | 74% | 37% |
End-to-end cascade
| Stage | $/paper, all stages | Known major errors still reported as major | …of them real (not seeded) | Precision (lower bound) |
|---|---|---|---|---|
| Screen: Luna low ∪ Haiku low (the scale-run screen) | $0.0056 | 292/460 (63%) | 120/225 | 51% |
| + Claude Haiku 5.5 high verifies | $0.0089 | 143/460 (31%) | 61/225 | 68% |
| + GPT-6 Luna high verifies (both must say major): the candidates | $0.0103 | 128/460 (28%) | 53/225 | 72% |
| + Opus 5.5 high adjudicates | $0.0382 | 110/460 (24%) | 45/225 | 76% |
| + GPT-6.1 Sol high adjudicates: the final list | $0.0501 | 108/460 (23%) | 44/225 | 78% |
| Alternative: GPT-6 Luna high as the only verifier | $0.0101 | 186/460 (40%) | 81/225 | 67% |
| Alternative: GPT-6.1 Sol medium as the only verifier | $0.0600 | 189/460 (41%) | 82/225 | 67% |
Pooled over the four benchmarks (seeded, PMC-version corrections, journal-corrected, verified real errors). Requiring both cheap verifiers to agree raises precision (72%, against 67% for Luna high alone and 67% for Sol medium) but keeps fewer known major errors: 128 of 460, against 186 for Luna high alone ($0.0101 per paper) and 189 for Sol medium ($0.0600). At scale the pair's job is to cut the stream of flags cheaply before the expensive judges.
On a uniform random sample of 120 scale-run candidates, Opus 5.5 (high) and GPT-6.1 Sol (high) both called 43% a real major error (95% CI 35%–52%) and 79% a real error of any severity (71%–85%). The judges differ mainly on severity: Sol alone calls 76% major, Opus alone 50%.
Pilot on 2,000 random high-impact papers
| Stage | Count | Cost |
|---|---|---|
| Papers screened (Haiku-low ∪ Luna-med) | 2,000 | $13.47 |
| Flags (all severities) | 10,446 | |
| Major flags (confidence ≥ 0.5) | 695 | |
| Luna-high verifier: confirmed major | 259 | $3.24 |
| GPT-6.1 Sol judge: confirmed major | 191 in 140 papers | $14.76 |
Total $31.47 = $0.0157 per paper.
Realistic precision on random papers (oracle = Opus 5.5 high and GPT-6.1 Sol high agreeing, 150 sampled flags, half decided by the oracle): Luna-high alone 0.56 · Luna-high → GPT-6.1 Sol judge 0.92 · Luna-high → Haiku-high (both say major) 0.97 at 6% of the judge's cost. Luna rejections lost 1 of 35 sampled majors. These figures count only flags on which the two oracle models agreed about severity, which overstates precision; the scale-run section gives an unbiased random-sample estimate.
Preregistration track: trials vs their ClinicalTrials.gov registration
7,184 trial reports screened by GPT-6 Luna (high) and Claude Haiku 5.5 (high); 947 where both flag an undisclosed major discrepancy went to Opus 5.5 (high), which confirmed 280 major and 554 minor (48 that Opus 5.5 refused were verified by Opus 5 instead).
Confirmed major discrepancies
| Journal | Paper / registration | Summary (Opus verifier) |
|---|---|---|
| BMC Med 2026 | PMC13081374 NCT05929183 | The paper states that the trial was prospectively registered (NCT05929183). The registry shows the record was first submitted on 26 June 2023 and posted on 3 July 2023. By the paper's own account, recruitment began in November 2022 and the study ran until 30 June 2023, so the registration was made essentially at the end of the trial and the prospective-registration claim is inaccurate. |
| JAMA Netw Open 2024 | PMC10980970 NCT05415085 | The paper states the trial was prospectively registered. ClinicalTrials.gov shows the registration was first submitted on 2022-04-27, about a year after enrollment began in April 2021 and near the end of recruitment. The registration was therefore retrospective, and the paper's prospective-registration claim is incorrect and undisclosed. |
| Diabetologia 2024 | PMC11604790 NCT06115460 | The ClinicalTrials.gov record lists change in time in range (TIR) at 3 months as the sole primary outcome and microalbuminuria as secondary. The publication instead names change in UACR as the primary outcome, powers the study on it, and lists TIR as secondary, without acknowledging the change. Both outcomes are reported and both favour sitagliptin, but this is an undisclosed swap of the primary outcome; the registration was also submitted retrospectively, after primary completion. |
| BMC Med 2025 | PMC12217910 NCT05827822 | The ClinicalTrials.gov record names average risky drinking days per month as the sole primary outcome, with PACS craving as a secondary outcome assessed at two weeks. The published report instead presents PACS craving from baseline to 8 weeks post-intervention as its primary outcome, and reports risky drinking days (non-significant, P = 0.13) as secondary. The paper does not acknowledge this change. |
| JAMA Netw Open 2026 | PMC12980234 NCT06276569 | The ClinicalTrials.gov record lists nine primary outcomes, including SpO2/FiO2, inflammatory and myocardial markers, APACHE II and Murray scores at days 1-7, length of stay, and 30- and 90-day mortality; it does not list ARDS incidence. The published report designates ARDS incidence as its sole primary outcome and recasts the registered primaries as secondary, without acknowledging any change. Several registered measures are reported differently or not at all: PaO2/FiO2 instead of SpO2/FiO2, 48-hour worst scores instead of days 1-7, and no 30-day mortality. |
| Cell Rep Med 2026 | PMC13522775 NCT05459441 | The ClinicalTrials.gov record lists clinical response rate (CR+PR at 24 months) as the sole primary outcome and adverse events as a secondary outcome. The publication instead names dose-limiting toxicity incidence as the primary endpoint and presents response as secondary or exploratory. The paper does not acknowledge or explain this change. Response rates are still reported, but they are framed as exploratory. |
| BMC Med 2018 | PMC6038204 NCT00700895 | The registration lists a single primary outcome \u2014 the number of warfarin dose titrations required to achieve target INR assessed over 3 months (compiled every 4 weeks) \u2014 whereas the published report designates the number of dose titrations in the first 2 weeks (day 14) as the primary efficacy endpoint, with the sample size, non-inferiority margin and abstract headline result all based on that 14-day window. The 3-month titration count is reported, but as a secondary outcome. The paper discloses a 2016 protocol amendment adding time-in-therapeutic-range as a secondary endpoint but does not mention or justify the change in the primary outcome's time frame, and the registry entry was not updated accordingly. |
| Circulation 2021 | PMC8849598 NCT03476785 | The paper states the trial was registered prospectively, but the ClinicalTrials.gov record (NCT03476785) was first submitted in March 2018, nearly three years after the listed study start of June 2015. The registered enrollment (56) matches the trial, and the paper does not acknowledge the late registration. The registered primary outcome (LV stiffness at 1 year) is consistent with what was reported. |
| Am J Public Health 2009 | PMC2692954 NCT00314028 | The ClinicalTrials.gov registration lists frequency of unprotected penile-vaginal sex at 3 months as the sole primary outcome. The publication instead names subsequent STD diagnosis over a 6-month record review as the primary outcome and reports unprotected sex as a secondary behavioural outcome, which was not significant in adjusted and imputed analyses. The paper does not acknowledge or explain this difference from the registration. |
| JAMA Netw Open 2018 | PMC6324427 NCT01767883 | The ClinicalTrials.gov record lists the primary outcome as the percentage of patients meeting guideline-concordant care goals (BP, LDL, HbA1c, ACE/ARB use, NSAID avoidance, nephrologist referral, smoking cessation). The published report instead names annualized eGFR loss as its primary outcome, a measure absent from the registration. The paper does not acknowledge this difference, and it reports only some components of the registered measure as separate secondary outcomes, with no composite result. |
| PLoS Med 2021 | PMC8423266 NCT03518151 | The ClinicalTrials.gov record (NCT03518151) lists women's dietary quality score at 3 months as the sole primary outcome, with store sales of fruit/vegetables and confectionery as secondary outcomes. The published paper designates store sales as the primary outcome and dietary measures as secondary, without acknowledging or explaining the change. The registered primary outcome is still reported, but it is presented as secondary. |
| JAMA Netw Open 2026 | PMC13539392 NCT06033365 | The paper states the trial was registered before enrollment began, but the registry shows it was first submitted on April 25, 2023, about two weeks after the paper's reported enrollment start of April 10, 2023, and was posted publicly in September 2023. The registered primary outcome was also narrower: outpatient addiction treatment attendance at 30 days. The paper's primary outcome is a composite measured at both 30 and 90 days, and this difference from the registration is not acknowledged, although both results were null. |
| JAMA Netw Open 2024 | PMC11031685 NCT04191837 | The trial registration lists NRS pain at week 4 as the sole primary outcome, with week 8 and week 12 as secondary time points. The published abstract instead designates NRS pain at 12 weeks as primary, without acknowledging the change. The registered week-4 result is reported and also statistically significant, so the main conclusion is not reversed, but the primary time point was changed without disclosure. |
| JAMA Netw Open 2024 | PMC11074813 NCT04260724 | The ClinicalTrials.gov record lists the 4-week change in ADAS-Cog (among nine 4-week cognitive measures) as primary and the 8-week ADAS-Cog change as secondary. The publication names the 8-week ADAS-Cog change as the primary outcome and the 4-week change as secondary, without noting or explaining the change. The 4-week result was also statistically significant, but the switch in primary time point is undisclosed. |
| BMC Med 2024 | PMC11229192 NCT05503641 | Three of the six primary outcomes registered for NCT05503641 (dynamometer handgrip strength, the Lawton & Brody IADL scale and the Barthel Index) do not appear in the published trial report. The paper gives no explanation or disclosure, and it reports several unregistered outcomes in their place. There are also inconsistencies in reported group sizes (98 randomised, 95 analysed, Table 2 headers showing n=26/24), though these are of lesser weight. |
| BMC Med 2024 | PMC11406833 NCT04972032 | The registration specifies the primary outcome as the 60-day adhesion recurrence rate, with recurrence defined as any AFS score above 0 at second-look hysteroscopy. The publication instead reports as primary the rate of 'adhesion reduction', defined as a decrease of at least 4 AFS points, and does not report the registered AFS>0 recurrence rate. This change of primary outcome definition is not acknowledged or explained in the paper. |
| Diabetes Care 2024 | PMC11664189 NCT04545411 | The ClinicalTrials.gov registration lists five primary outcomes, including two measures of vascular endothelial function and measured HbA1c change. The published report presents neither endothelial outcome and gives only a CGM-derived estimated HbA1c, without explaining the omissions. The paper also used 7 days of CGM rather than the registered 2-week window; this is a smaller, secondary inconsistency. |
| JAMA Netw Open 2025 | PMC11866027 NCT04292379 | The paper states the protocol was registered before participant enrollment. The ClinicalTrials.gov record was first submitted on 27 February 2020, after the 29 January 2020 start of the preimplementation enrollment period described in the paper itself. The registered primary outcome (30-day child well-being) matches the published one, so the issue is limited to the accuracy of the prospective-registration claim. |
| Clin Cancer Res 2025 | PMC12353064 NCT02494869 | The ClinicalTrials.gov record lists only one primary outcome: gene expression patterns and H&E morphological features of normal breast epithelium over 2 years. The published report instead names change in Ki67 proliferation at week 25 as its primary endpoint and does not report the registered outcome. The paper does not acknowledge or explain the difference, so this is an undisclosed primary outcome change. |
| JAMA Netw Open 2025 | PMC13102256 NCT03965195 | The ClinicalTrials.gov registration lists all-cause hospitalization as the sole primary outcome. The published report designates respiratory-related (ICD-10 J-code) hospitalization as primary and calls all-cause hospitalization a secondary outcome, without acknowledging the change. All-cause hospitalization results are still reported and were null, like the new primary outcome. |
Deterministic registry checks (dates and identifiers compared by code, then verified by Opus 5.5):
9 cases
| Journal | Paper / registration | Check | Verdict | Summary |
|---|---|---|---|---|
| JAMA Netw Open 2024 | PMC10980970 NCT05415085 | false prospective claim | major | The paper states that the trial was prospectively registered as NCT05415085. The registry shows it was first submitted on 2022-04-27, roughly a year after enrollment began in April–May 2021 and close to the primary completion date. The registration is therefore retrospective, and the paper's claim of prospective registration is inaccurate. |
| Eur Respir J 2025 | PMC12332466 NCT02526004 | false prospective claim | minor | The paper calls CFMATTERS 'prospectively registered' as NCT02526004, but the registry shows first submission in August 2015, after recruitment began in January 2015. The paper itself acknowledges the delay in registration, and the registered primary outcome matches the reported one. The wording is therefore inaccurate but transparently qualified, making this a minor issue. |
| BMC Med 2026 | PMC13081374 NCT05929183 | false prospective claim | major | The paper says the trial was prospectively registered as NCT05929183. The registry shows it was first submitted on 26 June 2023, whereas the paper reports recruitment from November 2022 to 30 June 2023. The registration was therefore made retrospectively, near the end of the study, and the paper's claim of prospective registration is inaccurate. |
| BMC Med 2012 | PMC3292842 NCT00981877 | registration mismatch | minor | The ClinicalTrials.gov number cited in this paper (NCT00981877) belongs to an unrelated trial of probiotics for rotavirus diarrhea in Bolivian children, not to this iron-supplementation trial in Swiss blood donors. This looks like a citation error rather than evidence of outcome switching. However, it means the published report cannot be checked against its actual registration through the cited ID. |
| Eur Respir J 2012 | PMC3432241 NCT00759304 | registration mismatch | minor | One of the two ClinicalTrials.gov identifiers cited in this nicotine mouth spray trial (NCT00759304) points to an unrelated observational cohort study of ageing and cardiovascular events in 65-year-olds. This appears to be a mis-cited registration number. The second cited identifier may be the correct registration, and the error does not affect the reported results. |
| Eur Respir J 2012 | PMC3432241 NCT00766584 | registration mismatch | minor | One of the two ClinicalTrials.gov identifiers cited by this nicotine mouth spray trial, NCT00766584, belongs to an unrelated French observational cohort on aging, autonomic activity and sleep apnea. This appears to be a citation error, since the paper also lists a second identifier (NCT00759304) that may be the correct registration. The error affects traceability but not the reported results. |
| Lancet Diabetes Endocrinol 2016 | PMC4728199 NCT00797862 | registration mismatch | major | The ClinicalTrials.gov number cited for PATHWAY-3 (NCT00797862) belongs to a different trial: a 1254-patient aliskiren/amlodipine study (ACCELERATE) with a blood-pressure primary outcome. PATHWAY-3 is a 441-patient amiloride/hydrochlorothiazide trial with a 2 h glucose primary outcome. The cited registration therefore cannot be used to verify the trial's prespecified outcomes or registration timing; only the EudraCT number cited alongside it may be valid. |
| Circulation 2017 | PMC5459585 NCT01858363 | false prospective claim | major | The paper states that the trial was prospectively registered as NCT01858363. The registry, however, shows a first submission on 8 May 2013, after the November 2012 study start, and the paper itself reports enrollment beginning in December 2012. Registration therefore occurred roughly 5 months after enrollment began, and the prospective-registration claim does not hold. |
| Circulation 2021 | PMC8849598 NCT03476785 | false prospective claim | major | The paper states the trial was registered prospectively on ClinicalTrials.gov (NCT03476785). The registry record was first submitted on 2018-03-01, nearly three years after the listed study start of 2015-06-01 and about 16 months before primary completion. This indicates retrospective registration, and the paper does not acknowledge it. |
Compared against the current registry record (earlier registry versions are not available to us), so a registry edited after publication can hide or create discrepancies. Secondary analyses and follow-up reports are recognised and not penalised for omitting the primary outcome.
Scale run: all 438,528 research articles in scope
| Screener | Papers screened | Major flags | Cost |
|---|---|---|---|
| haiku55-low-checklist | 438,528 | 124,881 | $1362.39 |
| luna6-low-checklist | 265,000 | 33,759 | $554.50 |
| luna6-med-checklist | 5,000 | 836 | $16.17 |
Cascade (final): 123,861 major flags → Haiku-high confirmed 21,111 → Luna-high confirmed 13,497 candidates (verification cost $487.43).
On-demand Bedrock (batch is not offered for these models). Complete as of 2026-10-23 15:50 UTC: Claude Haiku 5.5 read every paper in scope; GPT-6 Luna, limited by its throughput quota, read the first 260,000 in the same order as a second screener before the run reached its $4,500 cap (so those papers had both).
Adjudicated: 5726 major errors in 4482 papers
Top 9400 scale-run candidates re-checked independently by Opus 5.5 (high) and GPT-6.1 Sol (high; from 18:30 UTC only on what Opus calls major, since Opus is the stricter judge); 5726 (61%) were called a real major error by both. Not yet hand-verified. Opus 5.5 refused about 10% of these requests (stop reason 'refusal' on ordinary papers); 1158 were judged by Opus 5 (high) instead. On 50 candidates both answered, Opus 5 and Opus 5.5 agree on real vs not in 90% and on major vs not in 72%, with similar major rates (26 vs 28). Adversarial check (GPT-6.1 Sol builds the strongest honest defense of the authors, Opus rules) on a random 40 adjudicated majors: 33 (82%, 95% CI 68%–91%) upheld as major, 7 downgraded to minor, 0 judged not an error; on a random 40 from the most recent, lowest-ranked 2,000 adjudications: 30 (75%, 95% CI 60%–86%) upheld as major, 10 downgraded to minor, 0 judged not an error. Downgrades are mostly a slip whose correct value is unambiguous elsewhere in the paper. Independent agent audit (Claude agents with Python and web access; one reproduces the error from scratch, one argues the authors' side, one judges materiality and searches for published corrections) on a random 20: 20/20 reproduced as real errors, 13/20 (65%, 95% CI 43%–82%) also judged major and uncorrected; on the 20 highest-confidence: 20/20 reproduced as real errors, 18/20 (90%, 95% CI 70%–97%) also judged major and uncorrected, 1 already corrected by the journal. Published notices: of 4,084 flagged papers swept (PMC, Crossref, Europe PMC), 273 have a correction, retraction or expression of concern. For 42 confirmed findings the notice fixes the very error we flagged from the original text (both Haiku 5.5 and GPT-6 Luna agree) - independent confirmation by the journals themselves; those rows are marked as already corrected in the downloads. 142 notices concern something else, and 158 are too thin to tell. Author manuscripts: 1,639 of 5,726 confirmed findings (29%) are in papers whose PMC copy is the authors' accepted manuscript rather than the journal's final version. Errors can be fixed during copy-editing: in the write-up checks, 5 of 18 such errors were absent from the publisher's version of record, so roughly 8% of the list may not appear in the final published paper. The downloads flag these rows. Ordering experiment: since 22:30 UTC adjudication batches are split between a learned ranker (logistic regression on the cheap verifiers' confidences, flag category and text, retrained on all adjudications so far; before the experiment, 5-fold cross-validated AUC within the 2,950 top-ranked candidates adjudicated by then was 0.71 vs 0.61 for mean confidence, and on 66 random-sample candidates outside that set both orderings scored about 0.50, too few to separate them) and the cheap verifiers' mean confidence, half and half for the first batch, 85/15 for the next five, and all learned from 05:25 UTC. Both-major so far: baseline 263/645 (41%), learned 3327/5505 (60%) (Fisher exact p = 2e-21).
For comparison: On a uniform random sample of 120 scale-run candidates, Opus 5.5 (high) and GPT-6.1 Sol (high) both called 43% a real major error (95% CI 35%–52%) and 79% a real error of any severity (71%–85%). The judges differ mainly on severity: Sol alone calls 76% major, Opus alone 50%. Taking the top-ranked candidates first raises the both-major rate from 43% to 61%.
List
| Journal | PMCID | Category | Opus summary |
|---|---|---|---|
| BMC Med 2025 | PMC12211446 | ci | The confidence intervals for the headline prevalence in the abstract and results, and for every genotype-specific prevalence, are mathematically impossible for n = 822. The correct 95% CI for 6.2% is roughly 4.6%-7.9%, not 6.18%-6.22%. Because this sits in the abstract's headline result and greatly |
| Nat Commun 2025 | PMC12575868 | table_arithmetic | The nationwide savings figure is overstated by a factor of 1000. The stated inputs (26.72 MJ/m² × 97 billion ft²) give about 66.9 million MWh, not 66.87 billion MWh. The 66.87 billion MWh figure is also physically impossible, being far more than total U.S. electricity use. The same factor-of-1000 er |
| Nat Commun 2017 | PMC5309857 | p_value | Genuine error. The Fisher exact and χ² p-values in Table 1 cannot come from the tabulated mouse counts: recomputation gives two-sided p≈0.26 (reported 0.0006), p≈0.08 (reported <0.0001) and 3×2 χ² p≈0.15 (reported <0.0001). The reported values are exactly what one obtains by running Fisher's test on |
| Ann Rheum Dis 2011 | PMC3232181 | ci | Confirmed genuine error: the meta-analysis result for FCGR2A and malar rash is reported as OR 1.11 with 95% CI 1.17 to 1.33, an interval that excludes its own point estimate and is therefore mathematically impossible. Back-calculation from the reported p=0.0031 gives a plausible CI of about 1.04-1.1 |
| JAMA Netw Open 2025 | PMC12457976 | abstract_body_mismatch | The abstract attributes the overall-cohort HLOS estimates to the hypotension subgroup. The actual subgroup estimates are SAE -1.65 (P=.04) and observation -0.13 (P=.86, not significant). The abstract and conclusions therefore falsely claim significantly shorter LOS with observation in hypotensive pa |
| Clin Cancer Res 2012 | PMC3596113 | p_value | The reported Fisher exact P = 0.0005 for 1/6 vs 7/18 CR/CRi is impossible. The correct two-sided Fisher P is about 0.62 (one-sided about 0.32). The error turns a non-significant difference into an apparently highly significant one. The authors use it as statistical support for the dose-response clai |
| JAMA Netw Open 2020 | PMC8094425 | p_value | The bone-only metastasis OS subgroup is reported as HR 1.22 (95% CI 0.88-1.68) with P < .001. The CI includes 1, and the recomputed two-sided P is about 0.23. The result therefore is not statistically significant. This error appears in the abstract, and it directly supports the paper's stated conclu |
| JAMA Netw Open 2025 | PMC12406064 | p_value | The reported P < .001 cannot follow from the reported t129 = 0.71 (P ≈ .48) or from the heavily overlapping CIs. The between-group difference in HDRS-17 change is not statistically significant. Because the abstract and the Results present this secondary outcome as 'significantly' greater with MBC, t |
| Sci Adv 2023 | PMC9891701 | claim_vs_data | Genuine error confirmed. The abstract ('Depletion of pancreatic-specific TASc promoted the tumorigenesis of PDAC tumors') and the Introduction ('Depletion of TASc promoted the tumorigenesis of PDAC') state the opposite direction of the paper's own findings: AAV8-GFAP-DTA–mediated TASc depletion impr |
| Cell Rep 2026 | PMC13560472 | abstract_body_mismatch | Confirmed. The Summary abstract and the 'In brief' both state that high metformin concentrations 'upregulate G6PC expression ... to prevent glucose release' in IECs, whereas the Results, Discussion, Figure 6 title, and the paper's closing summary all report that metformin significantly DECREASES/dow |
| JAMA Netw Open 2020 | PMC8094425 | p_value | The bone-only metastasis OS subgroup has HR 1.22 with a 95% CI of 0.88-1.68, which includes 1 and implies p of about 0.23. Yet the paper reports P < .001 in both the abstract and the results. The Discussion and Conclusions also explicitly claim significantly prolonged OS in this subgroup. This is a |
| BMC Med 2025 | PMC12403310 | claim_vs_data | The paper's own Results, Table 2, Table 3 and Discussion show 216/376 (57.4%) trials were concordant and 160/376 (42.6%) discordant. The abstract and both Conclusions sections say 'over half (216/376, 57.4%) had discordance', which mislabels the concordant group and turns the headline finding around |
| JAMA 2010 | PMC3655326 | p_value | Fisher's exact test on the reported renal cancer counts (4/45 vs 6/155) gives a two-sided p of about 0.24, not 0.004. The renal cancer association is therefore not statistically significant. It is a headline claim in the abstract and conclusions and is used to recommend renal surveillance, so the er |
| Clin Cancer Res 2012 | PMC3596113 | claim_vs_data | The Fisher exact P for 1/6 vs 7/18 is about 0.62 two-sided (0.32 one-sided), not 0.0005. The reported value turns a non-significant difference into a highly significant one. The Discussion uses it as evidence of a dose-response relationship supporting SCH 900776's contribution, so this is a genuine |
| Hum Brain Mapp 2022 | PMC9057088 | table_arithmetic | Table 4's T and p columns contradict each other and the reported means and SDs. - Large t values (8.6–12.9) are paired with p-values up to .995, which is impossible. - Recomputed t values for temporal, TH and whole brain are all below 1. - The abstract's claim that frontal volume is lower in ADHD (r |
| JAMA Netw Open 2026 | PMC13434869 | ci | The risk-difference CIs for the primary outcome, reported in both the abstract and the Results, are grossly too narrow. Recomputed from Table 2 counts, the correct 95% CIs are about 0.8–8.5, 5.7–13.0 and 1.2–8.1 pp. The paper's own z statistics confirm SEs of about 1.8–2.0 pp. The NNT CIs share the |
| Nat Commun 2025 | PMC12575868 | table_arithmetic | The stated inputs (9.01e9 m² × 26.72 MJ/m²) give about 66.9 million MWh, not 66.87 billion MWh. This is a 1000-fold unit error. It propagates to the $7.35 trillion cost and 66.94 billion t CO2 figures, all of which appear in the abstract and discussion as headline results, and the claimed total far |
| Nat Commun 2017 | PMC5309857 | p_value | Genuine error. The Fisher exact p-values in Table 1 cannot come from the tabulated mouse counts: 9/16 vs 4/13 positive gives two-tailed Fisher p = 0.26 (chi-square p = 0.17), not 0.0006; 4/13 vs 13/20 gives p ~0.08, not <0.0001; and the 3x2 chi-square gives p = 0.15, not <0.0001. The reported values |
| J Hepatol 2020 | PMC7395653 | ci | The reported 95% CIs are implausibly narrow and inconsistent with the stated Poisson assumption, and this applies to both the SIR and the SMR. Correctly computed, the SIR CI is about 157–230: the result stays significant, but the abstract's interval is wrong. The SMR CI with 20 deaths is about 58–14 |
| Lancet Reg Health Eur 2025 | PMC11910110 | abstract_body_mismatch | The error is genuine. The abstract's sentence on preterm birth reports low-birth-weight estimates from Table 4 for ADHD (4·15), ASD (3·23) and non-verbal IQ (-5·44). For ASD this turns a null preterm association (0·60, CI -2·60 to 3·79) into a stated significant one, contradicting the Results ('Not |
| Lancet HIV 2021 | PMC7869891 | ci | The Wilson upper 95% bound for 0/384 DTG-exposed outcomes is about 0.0099, not the reported 0.0010. The EFV bound (0.0036 for 0/1068) checks out under the same method, which confirms that the DTG value is wrong. This is the headline NTD estimate and it appears in the abstract. The wrong value unders |
| BMC Med 2023 | PMC10642059 | claim_vs_data | The abstract says all PPD-related variables were significantly associated with CVD mortality. Table 2 and the Results text show the opposite: no PPD measure was significantly associated with CVD mortality (P=0.275, 0.302, 0.337). This is a direct reversal of a result stated in the abstract, so it is |
| BMJ Glob Health 2025 | PMC11749663 | claim_vs_data | The RR of 2.55 (0.82 to 4.27) for complete abortion is mathematically impossible. The control risk is 96.1%, and Table 2's own figures (96.1% vs 97.3%, RD 1.49%) imply an RR of about 1.01–1.02. The CI is also symmetric on the arithmetic scale, which is inconsistent with a ratio. The erroneous value |
| Diabetes Care 2009 | PMC2732170 | table_arithmetic | The VA extrapolated costs in Table 3 were computed with the 75% population share instead of the stated 5%; the MCLNO and TUHC rows reproduce exactly with their stated shares. This inflates the VA cost from about $17M to $260M. The headline population lifetime cost in the abstract is therefore about |
| BMC Med 2025 | PMC12211446 | ci | All reported prevalence 95% CIs, including the headline 6.2% (6.18–6.22) in the abstract, are impossibly narrow. With n = 822, the correct binomial intervals span several percentage points (e.g. 6.2%: about 4.6–7.9%; 4.14%: about 2.8–5.5%). The paper therefore grossly misstates the precision of its |
| JAMA Netw Open 2026 | PMC13003376 | abstract_body_mismatch | The abstract says higher MELD was associated with 'lower referral odds', yet it reports OR 1.13 (95% CI 1.07-1.18). The Results, Table 2 and the descriptive data all show higher MELD was associated with increased odds of referral. This is a genuine direction reversal of a primary-outcome result in t |
| Nat Commun 2020 | PMC7679396 | ci | The repositive-rate CI (0.423–0.574%) does not contain the point estimate of 0.31% (107/34,424). No standard method could produce it; the correct exact CI is about 0.25–0.37%. The error appears in both the abstract and the Results, so it is a mathematically impossible reported key result in the abst |
| Proc Natl Acad Sci U S A 2022 | PMC8916005 | p_value | Genuine error. The reported chi-square P = 0.0379 is incompatible with the paper's own counts (19/45 vs 10/35), for which Pearson chi-square = 1.59, P = 0.21 (Yates P = 0.30; Fisher two-tailed P = 0.23). The reported value is exactly recoverable by running the test on the percentages as if each grou |
| Sci Adv 2026 | PMC12757054 | table_arithmetic | The chi-square statistics in Table 1 (and the analogous test in the Results text) cannot be derived from the reported animal counts: 4/16 vs 14/28 gives χ² = 2.63 (p ≈ 0.105), not χ² = 13.333 (P = 0.0003); 2/16 vs 9/28 gives χ² = 2.10 (p ≈ 0.15), not 10.351 (P = 0.0013); 4/16 vs 9/24 gives χ² = 0.68 |
| Nat Commun 2026 | PMC13351079 | claim_vs_data | The paper's Results report temperature as a significant NEGATIVE predictor of H7 diffusion at both origin (β = -0.484, 95% HPD -0.900 to -0.237) and destination (β = -0.227, -0.499 to -0.047), and the Discussion explicitly describes and mechanistically explains a negative temperature-transmission re |
| Hum Brain Mapp 2009 | PMC2903209 | p_value | The paired t statistics (-1.200, -1.495, -0.274, +0.175) match the Table I means and SDs. However, the corresponding p-values (0.001, 0.009) are impossible for these t values. The correct p-values are roughly 0.17–0.86, all non-significant. The p-values appear to have been copied from the baseline/f |
| Cell 2010 | PMC2887341 | abstract_body_mismatch | Genuine inconsistency: the abstract states that 'G3249A and G3244A' interfere with key sequence-recognition interactions and eliminate termination, whereas the Introduction, Results and Conclusions all consistently identify G3242A (not G3244A) as the second such mutation, and explicitly describe G32 |
| PLoS Pathog 2012 | PMC3261917 | abstract_body_mismatch | Genuine internal contradiction: the abstract claims arg1 knockdown 'led to increased generation of nitric oxide', whereas the Results heading, Results text, Fig. 6I legend, and Discussion all state that arg1 knockdown did NOT increase NO production, a point the authors rely on to argue a polyamine-m |
| Hepatology 2009 | PMC2759707 | claim_vs_data | Genuine and verified contradiction: the abstract states TIMP-1 and TIMP-2 increased after chronic CCl4 'only in the gp91phox-/- mice', whereas the Results (Fig. 7, p<0.05), the Discussion, and the paper's concluding mechanistic argument all state the increase occurred only in the WT mice and that gp |
| Int J Epidemiol 2018 | PMC6422062 | abstract_body_mismatch | The abstract says LBW prevalence rose from 9.0% to 10.1%. Table 1 data, which I recomputed, give 8.2% in 1982 and 8.3% in 2015. The Results, Discussion and Key Messages all say LBW stayed stable (P for trend 0.80). The abstract therefore reports a rising trend where the paper's data show none, which |
| BMJ Glob Health 2020 | PMC7061886 | claim_vs_data | The abstract and Results report OR 1.6 with a CI excluding 1 and p<0.001, yet state there was no statistically significant change in CFR. That is a direct self-contradiction in the headline result and conclusion. The OR is also inconsistent with the paper's own data: the CFR declined, so the OR shou |
| PLoS Comput Biol 2020 | PMC7048302 | claim_vs_data | Genuine error. Table 3 shows LabMol-149 with EC50 = 1.450 \u03bcM (3D7) and 0.509 \u03bcM (W2), i.e. 1450 nM and 509 nM, yet the Abstract and Discussion claim it shows "low nanomolar" activity (EC50 < 500 nM) and that it "completely satisfies" the antimalarial candidate criteria. No innocent explana |
| Psychol Med 2015 | PMC4682193 | claim_vs_data | The abstract calls the primary-outcome reduction in suicidal behaviour 'significantly greater'. The reported result is p=0.162 with a 95% CI (-1.71 to 0.09) that crosses zero, which is non-significant even at the trial's own lenient two-sided alpha of 0.15. The abstract also mislabels the raw treatm |
| Clin Cancer Res 2008 | PMC2907116 | ci | The error is genuine. Both reported hazard-ratio confidence intervals (2.0-3.1 for HR 2.5; 4.1-4.8 for HR 4.5) imply test statistics of z~8 and z~37, utterly incompatible with the accompanying p-values of 0.026 and 0.046; back-calculation from the p-values gives CIs of roughly 1.1-5.6 and 1.0-19.8. |
| Nat Commun 2018 | PMC5998138 | claim_vs_data | The abstract claims reduced PKN1 and PKN4 in CFBE41o- cells. The results show increased PKN2, confirmed by Western blot, and only ambiguous evidence for PKN1; PKN4 never appears. This is a headline abstract claim that contradicts the results in both direction and protein identity, so it is a major e |
| JAMA Netw Open 2025 | PMC12406064 | claim_vs_data | The reported t129 = 0.71 is incompatible with P < .001; it corresponds to P ≈ 0.48. The heavily overlapping CIs and the 1.1-point difference confirm the comparison is nonsignificant (P ≈ 0.3–0.5). The abstract and Results nonetheless call the HDRS-17 reduction 'significantly' greater, and this claim |
| Proc Natl Acad Sci U S A 2022 | PMC8916005 | p_value | The error is genuine and reproducible. For 19/45 (42%) vs 10/35 (28%), the 2x2 Pearson χ² is 1.59 (P = 0.21; Yates P = 0.31; Fisher P ≈ 0.23), not P = 0.0379. The published value is exactly what one obtains by applying χ² to the rounded percentages as if they were counts out of 100 (χ² = 4.31, P = 0 |
| Lancet Reg Health Eur 2025 | PMC12859599 | abstract_body_mismatch | The abstract presents the univariate HRs as the adjusted multivariate result and assigns them to the wrong treatments (immunotherapy 0.470 vs the body's targeted 0.470). The body and Discussion give adjusted HRs of 0.768 for immunotherapy and 0.556 for targeted therapy. The direction of the conclusi |
| PLoS Comput Biol 2025 | PMC11975116 | abstract_body_mismatch | The abstract's headline claim that WGS spanned '49 (98%)' of the 48 P-VNTRs is impossible. It also contradicts the Results (92%) and the Discussion (45 of 48, matching the three named poorly covered loci). This is a real abstract–body mismatch in a headline comparison of the paper. The qualitative c |
| Neurology 2026 | PMC13445500 | abstract_body_mismatch | The abstract reports that 73.5% (175/238) of trials used at least one biomarker. The body (Results, the phase sums 79+55, and the Discussion) consistently gives 56.3% (134/238). The abstract figure appears to be the clinical-outcome-as-primary statistic copied into the wrong place. This is a genuine |
| Psychol Med 2026 | PMC12902173 | sample_size | The analytic sample sizes conflict irreconcilably. Table 2 and the text give 361,258 participants for Cohort 2, while the flowchart gives 32,220, and the reported event percentages match only the flowchart. Cohort 1 shows the same pattern (468,722 vs 36,804), and its 'prior IBS excluded' cohort cont |
| Nat Commun 2023 | PMC9990026 | p_value | The flag is correct. The headline acquisition result (3/7 vaccinated vs 0/6 control uninfected after 9 challenges) gives a one-sided Fisher exact p of 35/286 = 0.12 (0.091 if the figure's n=6 vaccinated is used), i.e., non-significant. The reported 'P < 0.0001' is not merely wrong but unattainable b |
| PLoS Pathog 2023 | PMC10449197 | abstract_body_mismatch | Genuine error. All internal evidence (Beta = +1.00 for B.1.177, Table 1's 2.15 factor, the Results section heading, the sensitivity analysis, and the Discussion's '115% higher among Alpha') shows Alpha had ~2.15-fold (115%) HIGHER viral burden than B.1.177, equivalent to B.1.177 being ~54% lower. Th |
| Commun Biol 2025 | PMC12855784 | other | The error is genuine and not explainable by rounding, alternative settings, or subgroup/denominator differences. Table 1 (P41212, 28.81–1.45 Å, 1 molecule per ASU, 733 macromolecular non-H atoms, Wilson B 20.9, average B 23.55) is internally self-consistent with a single ~115-residue chain in a tetr |
| Eur Respir J 2023 | PMC10600351 | abstract_body_mismatch | The abstract reports p<0.001 for the CF versus rheumatoid arthritis matched comparison. Table 4 gives p=0.055, and the reported HR and CI (1.21, 1.00–1.48) imply p≈0.056. The abstract p-value is wrong, likely copied from the unadjusted column. The text's claim that CF risk was 'greater than' RA is n |
| Nat Commun 2019 | PMC6520353 | claim_vs_data | Genuine error: the abstract swaps the two human heritability estimates, stating that host variation explains 'almost half' of susceptibility and 'one-third' of severity, whereas the Results, Table 4 and Discussion consistently report 29% \u00b1 7% for susceptibility and 49% \u00b1 14% for severity ( |
| Proc Natl Acad Sci U S A 2018 | PMC6176606 | claim_vs_data | Genuine internal contradiction. The abstract states that the NS3-G459L variant 'dominantly inhibited negative-strand RNA synthesis' of WT virus, whereas the Results explicitly report that negative-strand replicon RNA was significantly above the GDD baseline (negative-strand synthesis permitted) whil |
| Kidney Int 2011 | PMC3394593 | claim_vs_data | Genuine error confirmed. The abstract sentence 'Only TLR4 (-/-) leukocytes produced IL6 in vivo and in response to HMGB1 in vitro' reverses the genotype: the Results (Figures 9 and 10), both figure legends, and three separate Discussion points all state that only TLR4 (+/+) leukocytes/macrophages pr |
| JAMA Netw Open 2023 | PMC9856961 | ci | This is a genuine error. The headline HbA1c effect of 0.02% lies outside its own reported 95% CI (0.03-0.14), in both the abstract and Table 3. The CI and the raw changes point to an effect of roughly 0.09 percentage points, and the reported SE and p value are also inconsistent with the estimate. Be |
| J Neurosci 2010 | PMC2874744 | abstract_body_mismatch | The flagged inconsistency is genuine. The abstract states that gain-of-miR-15a function reduces and loss-of-miR-15a function increases OGD-induced CEC death ('respectively'), which is the exact opposite of the Results (Fig 5A–D: mimic aggravates death and caspase-3 activation; inhibitor alleviates t |
| Kidney Int 2021 | PMC7927648 | ci | The point estimate 8.4 lies outside its own reported 95% CI (4.4–6.7), which is mathematically impossible. The same values appear in the abstract, Results and Table 4. This is a genuine error, most likely a mistyped upper bound. Significance is probably not overturned, but the CI for a headline resu |
| Gastroenterology 2020 | PMC7857017 | claim_vs_data | Genuine error. The abstract states 'Lower DDIT4 expression correlated with longer survival of gastric cancer patients,' which is the reverse of what is reported everywhere else in the paper (Introduction: 'Patients with low expression have poorer survival'; Results: 'statistically significant correl |
| JAMA Psychiatry 2015 | PMC5103694 | ci | The reported OR of 0.96 lies outside its own 95% CI of 0.01–0.75 and is inconsistent with P = .03. The CI and P instead imply an OR of about 0.09. The figure is impossible as reported and appears in the abstract and in the headline predictive result. It looks like a transcription error, since the co |
| JAMA Netw Open 2024 | PMC11579790 | claim_vs_data | The Key Points and abstract Conclusions state that JAKis had a statistically significantly increased cancer risk versus TNFis. The reported HR of 1.36 (95% CI, 0.94-1.96) includes 1, and the paper's own Results, Discussion and main Conclusions say the JAKi association was not significant. This misst |
| Lancet Reg Health Eur 2025 | PMC11910110 | abstract_body_mismatch | The abstract attributes the low birth weight estimates for ADHD, ASD and non-verbal IQ to preterm birth. For ASD this reverses the result: the body states preterm birth was not associated with ASD (0.60, CI crossing 0), yet the abstract reports a significant preterm–ASD association. The externalizin |
| Eur Respir J 2009 | PMC5516636 | p_value | The flag is correct and in fact understates the problem: the 95% CI column of Table 3 (the paper's primary results table) is systematically corrupted. In roughly 20 of 36 rows the reported β lies outside its own CI, or the CI's relation to zero contradicts the reported p-value (e.g. rs1171279 β=-0.0 |
| Sci Adv 2021 | PMC8357232 | abstract_body_mismatch | Confirmed: the abstract states CCW702 was 'well tolerated up to 34.1 mg/kg per dose' in cynomolgus monkeys, whereas the Results, Fig. 6 legend, and Methods consistently report the top dose/NOAEL as 34.1 \u03bcg/kg (dose groups 2, 9.8, 34.1 \u03bcg/kg; detectability \u22659.8 \u03bcg/kg; exposure 1.0 |
| J Hepatol 2020 | PMC7395653 | ci | With 20 observed deaths, a Poisson-based 95% CI for an SMR of 95.55 is about 58–148. The reported 94.25–96.85 is mathematically implausible. The corrected interval includes 100, so the claim in the abstract, highlights and conclusions that LT patients have lower mortality than the matched general po |
| BMJ Glob Health 2022 | PMC9494011 | ci | The abstract reports a headline association (queer orientation and healthcare provider discrimination) as significant with p=0.002, but its own 95% CI of 0.86–7.36 includes 1. Recomputing from the CI gives p ≈ 0.09. The table's ** marker and the results text repeat the same false claim of significan |
| Commun Biol 2025 | PMC12855784 | impossible_value | The flag is correct and is more extensive than reported. Table 1 (P4(1)2(1)2, 1 molecule/ASU, 1.45 A, 733 macromolecular atoms, average B = 23.55 A^2) is mutually incompatible with the Results text and figures (C222(1), four molecules MolA-MolD in the ASU forming AB and CD dimers, 1.98 A, average B |
| J Clin Oncol 2024 | PMC11404760 | ci | The abstract reports HR 0.12 with 95% CI 0.14–1.13 and P = .03. This is mathematically impossible: the estimate lies outside the interval, and an interval spanning 1 cannot give P < .05. It also contradicts the internally consistent body values (HR 0.10, 95% CI 0.01–0.96, P = .046). The error sits i |
| Lancet Reg Health Am 2026 | PMC12811418 | abstract_body_mismatch | The headline result is inconsistent in four places. The abstract gives RR 88.82 ('89 times'), while Table 3 gives 83.82 with the same CI, so the abstract value is a likely typo. The Results text quotes 18–29 subgroup values as if they were the overall model, and the Discussion cites an unsupported 3 |
| Lancet Reg Health Am 2026 | PMC13382474 | abstract_body_mismatch | All three effect sizes in the abstract (3.25%, 5.92%, -0.30%) disagree with both Table 2 and the Results text (2.13%, 4.41%, -0.25%). The Table 2 coefficients reproduce the Results-text values exactly, and the abstract numbers cannot be derived from any reported estimate. Direction and significance |
| Psychol Med 2026 | PMC12902173 | sample_size | This is a genuine and large inconsistency in the core sample size. The text and Table 1 report 234,361 per matched group (468,722 total). The flowchart reports 18,402 per group (36,804 total), and the incidence percentages match the flowchart denominators. The 468k figure is also implausible within |
| Diabetologia 2025 | PMC12686032 | abstract_body_mismatch | Genuine abstract–body inconsistency. The abstract states that at seroconversion 'both FDR+ and FDR- FDR+ children were significantly more often positive for GADA and multiple autoantibodies' than FDR- children, whereas Table 3 and the Results and Discussion sections state and show no significant dif |
| Cancer Res 2009 | PMC2644342 | abstract_body_mismatch | Verified. Table 3 is internally consistent (recomputing crude ORs from the reported counts: IVS11+931A>G AA = (294/274)/(394/318) = 0.87, AG+GG = (144/98)/(129/126) = 1.44; IVS11-896A>T TT = (137/147)/(192/132) = 0.64, AT+AA = (303/235)/(331/310) = 1.21). The abstract's second pair (0.64, 0.46-0.88 |
| Cancer Res 2008 | PMC2849650 | claim_vs_data | Verified. The text reporting Fig. 3B gives NCI-H661 negative control 97.0 ± 7 versus CYGB-shRNA 35 ± 3 while asserting a 'significantly increased number of colonies as a result of CYGB knockdown' — the printed means show a ~64% decrease, directly contradicting the sentence, the abstract ('resulting |
| BMC Med 2019 | PMC6878715 | abstract_body_mismatch | The flag is confirmed and is not explainable by adjustment, population, or rounding differences. The abstract reports age-group adjusted prevalence ratios of 1.97 (1.70–2.28) for 0–6 months and 1.72 (1.51–1.95) for 6–12 months, whereas the Results text and Table 2 report 1.41 (1.22–1.51) and 1.36 (1 |
| Gastroenterology 2012 | PMC3978718 | abstract_body_mismatch | Genuine internal inconsistency: the abstract reports a 31% reduction in plasma endotoxin activity on the prudent-style diet while the Results text reports a 38% decrease for the same measurement in the same 8 subjects. The matching +71% Western-diet value in both places rules out a different populat |
| JAMA Netw Open 2021 | PMC8042532 | abstract_body_mismatch | This is a genuine error. The abstract's headline Key Points finding, repeated in the abstract Results, states that 5-year mortality for adenoid cystic carcinoma was 8.33%. Table 1 and the body Results give 8.33% for medullary carcinoma and 11.61% for adenoid cystic carcinoma. The Discussion repeats |
| Psychol Med 2021 | PMC9693673 | impossible_value | The logistic regression results labelled 'OR' are clearly log-odds coefficients: negative 'ORs' and negative CI bounds are impossible. This is not a single typo. It runs through every logistic result and the abstract's headline figures, which understate the true odds ratios several-fold (e.g. about |
| Hum Brain Mapp 2022 | PMC9057088 | table_arithmetic | From the reported means and SDs, the frontal volume difference gives t ≈ 1.41 (p ≈ 0.16), not t = 12.871 / p = .008. The t and p are also mutually inconsistent, and the method used reproduces the hippocampus t well, so the discrepancy is not a method artefact. The reported data therefore do not supp |
| Lancet Reg Health Eur 2025 | PMC12266187 | claim_vs_data | The headline 90-day mortality rate (63.7%, n=707) contradicts every stratified breakdown. These consistently give 408 deaths (36.8%) and about 701 survivors (63.2%), so the figure appears to be survival mislabelled as mortality. The misstatement sits in the abstract and describes the primary outcome |
| eLife 2024 | PMC10942632 | p_value | The reported p=0.0245 is actually P(X>=13), an off-by-one tail error. The correct one-sided p for 12 of 17 is 0.072, so the claimed 'significant excess' is not significant at 0.05. The same error appears in the second binomial test (reported 0.0012, correct about 0.0088), though that one remains sig |
| Lancet Reg Health Eur 2025 | PMC11910110 | abstract_body_mismatch | This is a genuine error. The abstract attributes the low-birth-weight ADHD estimate (4·15) to preterm birth, whereas the preterm estimate is 2·51. The mislabeling extends to ASD and non-verbal IQ in the same sentence. For ASD, the abstract claims a significant preterm association that the body expli |
| Clin Cancer Res 2025 | PMC12163594 | claim_vs_data | The abstract reports fatigue at 28.6% and 66.7% (escalation) and injection-site pain at 31.7% (expansion) as the most common grade ≥3 treatment-related AEs. These are actually all-grade, all-causality TEAE rates. Table 2 and the text show 0 grade ≥3 fatigue in both escalation cohorts and no grade ≥3 |
| JAMA Netw Open 2025 | PMC12188365 | p_value | The abstract states that the female share among additional eligible lung cancer cases (IPN-Potter Extra, 70/128) differed significantly from USPSTF 2021 (55% vs 48%, P < .001). The Results correctly report 55% vs 50% (370/747), P = .28, and recomputation confirms P ≈ .28. The abstract evidently copi |
| BMJ Glob Health 2022 | PMC9791420 | claim_vs_data | The abstract attributes the whole-cohort (Table 1) estimates to sexual minority adolescents. In the actual sexual minority analysis (Table 2), HIV-positive status is not significantly associated with anxiety (p=0.073), and unknown status is not significantly associated with suicidal ideation (p=0.06 |
| Gastroenterology 2017 | PMC5669829 | claim_vs_data | The abstract reverses the direction of the obesity, diabetes and metabolic syndrome differences. Table 1, the results text and the Table 5 diabetes OR all show these conditions were more common in nondrinkers. This is a genuine directional misstatement in the abstract, on a point central to the pape |
| JAMA Netw Open 2020 | PMC7265099 | claim_vs_data | The paper repeatedly states that HR 0.59 makes navigator children '69% less likely' to be hospitalized. The correct figure is about a 41% reduction in hazard, and the paper's own RR of 0.60 gives about 40%. The 69% apparently comes from inverting the HR (1/0.59 ≈ 1.69) and then misreporting it as a |
| J Hepatol 2010 | PMC6022746 | abstract_body_mismatch | Confirmed. The Abstract swaps the exposure labels for the single-factor cells of Table 4 in both interaction blocks: it assigns the non-significant ORs (2.01 and 1.20) to mutation-only subjects and the significant ORs (4.26 and 2.28) to high-AFB1-only subjects, whereas Table 4 and the Results text s |
| JAMA Netw Open 2022 | PMC9284328 | abstract_body_mismatch | The overall count of 46 592 (29.7%) for potentially unnecessary thyroid/insulin testing cannot be reconciled with the payer-specific counts (12 834 + 23 198 = 36 032, which is 23.0%). The Key Points report 23.0%, while the abstract results, main Results, Table 2 and Discussion report 29.7%. This is |
| Sci Adv 2024 | PMC11566992 | impossible_value | The flagged error is genuine. With 5.7% effective porosity and 0.65 M (or 6.5 M) CO2, the capacity is about 1.6 Mt/km3 (or 16 Mt/km3), not 1.5 Gt/km3 (or 15 Gt/km3). The reported values are physically impossible, since they exceed the mass of the rock itself, and look like a Gt/Mt unit error, probab |
| J Exp Psychol Gen 2019 | PMC6461371 | p_value | Both reported binomial p-values equal the point probabilities P(X=25)=.037 and P(X=26)=.021, not tail probabilities. The correct two-sided p-values are .154 for the Contingency condition (25/40) and .081 for the Frequency condition (26/40). Neither condition is significant at the paper's stated two- |
| Nat Commun 2025 | PMC12868887 | claim_vs_data | The paper reports p < 0.01 by Fisher's exact test for 18/20 vs 3/6 patency. Recomputation gives Fisher's exact p = 0.0624 (one- and two-sided identical for this table); even the most liberal alternatives (uncorrected chi-square p = 0.029, one-sided z p = 0.015) do not reach p < 0.01. The claimed p i |
| Diabetologia 2024 | PMC11732930 | p_value | The reported p=0.007 for hypertension and knowledge cannot follow from the stated means, SDs and group sizes. A t test on these numbers gives t ≈ 0.5 and p ≈ 0.6, so the result is not significant. The figure appears in the abstract and is presented as a significant association, so the error flips th |
| Mol Psychiatry 2016 | PMC5078854 | claim_vs_data | The abstract says cerebellar IDH3A protein is higher in BD than in controls. Table 3B (0.804 vs 1.227, P<0.05), the Results text and the Discussion all report it as significantly lower. This is a genuine direction reversal located in the abstract. Under the rubric (direction flip in the abstract) it |
| Stroke 2010 | PMC3016514 | abstract_body_mismatch | The abstract's headline effect sizes for recurrent vascular events are actually the all-cause death estimates from Table 3. The correct recurrence estimates (Table 2 and Results text) are 1.33, 1.06 and 1.20. Significance is unchanged, but this error sits in the abstract's headline result and overst |
| JAMA Netw Open 2025 | PMC12188365 | abstract_body_mismatch | The abstract states a statistically significant sex difference (55% vs 48%, P < .001) among additional Potter-eligible lung cancer cases. The Results, confirmed by recomputation, show this comparison is nonsignificant (55% vs 50%, P = .28). The abstract evidently copied the comparator and P value fr |
| Am J Psychiatry 2016 | PMC5026387 | claim_vs_data | The abstract states that 24.3% of people with lifetime cannabis use disorder received treatment. That figure is the severe-only lifetime rate; the overall lifetime rate in Table 5 and the Results text is 13.7%. This misreports a headline service-utilization result in the abstract, overstating treatm |
| Nat Commun 2017 | PMC5458555 | abstract_body_mismatch | The abstract's assignment of conformational preference is inverted with respect to the entire body of the paper. The Results define high FRET = closed and low FRET = open, report the high-FRET (closed) state as dominant (~70%) in apo, GDP·AlF4- and GDP states, and show GMPPCP (GTP mimic) shifting to |
| Nat Med 2023 | PMC10719106 | pct_count | The abstract's verified-vaccination percentages are arithmetically inconsistent with its own counts. The counts 355/1,058 and 439/1,544 give 33.6% and 28.4%, matching Table 2 and the main Results (difference 5.2%, CI 1.5–8.7, P = 0.005). The abstract instead reports 36.6% vs 30.3%, difference 6.3% ( |
| PLoS Med 2022 | PMC9725126 | abstract_body_mismatch | This is a genuine error. The abstract labels the 39.65 million S4 prevented CVD events as deaths, and attaches the 13.86 million S4 deaths figure to the test-accuracy scenario. The Results and the author summary state 39.65M CVD events (14.2%) and 13.86M deaths (13.7%). The headline mortality claim |
| J Clin Oncol 2024 | PMC11404760 | ci | The abstract reports HR 0.12 with 95% CI 0.14–1.13 and P = .03, for SLiM criteria predicting progression to MM. The point estimate falls outside its own interval, and the interval includes 1 despite P = .03. It also contradicts the self-consistent Results value of HR 0.10 (0.01–0.96), P = .046. This |
| PLoS Pathog 2018 | PMC5927469 | claim_vs_data | Genuine directional contradiction. The Results report a 15% INCREASE in colony-forming efficiency (i.e. greater survival/resistance) for csA mutant spores after 75\u00b0C treatment, and the Discussion states csA mutant spores 'were more resistant to heat'. The abstract, however, states that absence |
Unverified candidates (13,688 flags in 11,030 papers)
Two-model agreement only (cheap cascade); not yet independently checked. Categories: claim_vs_data 5544 · abstract_body_mismatch 4977 · p_value 717 · ci 639 · table_arithmetic 594 · sample_size 399 · impossible_value 322 · pct_count 244 · other 223 · outcome_switch 28 · label 1.
Top 40 by confidence
| Journal | PMCID | Category | Problem (model summary) |
|---|---|---|---|
| PLoS Med 2022 | PMC9718399 | pct_count | The unadjusted rates are labelled as percentages, but the stated counts give rates of about 26.4 and 20.1 per 1,000, not per cent. Table 4 reports the same raw rates as 'Rate per 1,000 patients at risk'. The abstract's u |
| Lancet 2012 | PMC10782811 | abstract_body_mismatch | The abstract reports five states (11%) with weights below 0.01, but Table 2 contains nine state estimates below 0.01. The reported count and percentage are also inconsistent with the stated total of 220 states. |
| Clin Cancer Res 2008 | PMC2830553 | abstract_body_mismatch | Abstract states 3% of patient samples had single-locus MSI, but results show 8 of 52 samples (about 15%) were L-MSI; the 3% refers to markers. |
| Gastroenterology 2012 | PMC3572711 | ci | The 95% CI for rs3802842 does not contain its point estimate, and its upper bound is almost equal to its lower bound. This SNP is one of the eight headline associations listed in the abstract and described as very strong |
| Psychol Med 2013 | PMC3936825 | abstract_body_mismatch | The abstract reports age standard deviations that conflict with Table 1. The bipolar subgroups’ means and SDs imply an overall patient SD of about 6.0 years, not 11.8; the control SD is reported as 6.1, not 12.0. |
| Hepatology 2008 | PMC2768766 | p_value | The overall-survival Wald statistic and p-value are inconsistent, so the reported significance of XIAP as an independent predictor of overall survival is not supported by the stated test. |
| Sci Adv 2024 | PMC10776005 | sample_size | Abstract states 149 TB subjects, while Results and Methods state 103 TB subjects (149 is the total of TB plus non-TB). |
| Lancet Public Health 2025 | PMC12700172 | claim_vs_data | The abstract says the unadjusted no-debt difference was not significant, but the Results and Table 3 report unadjusted HR 1.48 with CI 1.07–2.05 excluding 1, i.e. significant. Also the crude IRR in Table 2 is 1.52 (1.21– |
| JAMA Netw Open 2023 | PMC9982699 | pct_count | Incident IS count in abstract (15.5 million) is inconsistent with 3.4 million total incident strokes and with the 86.8% share; also 3458 of 4449 IS cases is 77.7%, not 86.8%. |
| JAMA Netw Open 2023 | PMC9982699 | abstract_body_mismatch | The subtype estimates are described as incident strokes, although their total and percentages correspond to the 17.8 million prevalent cases, not the 3.4 million incident cases. |
| PLoS Med 2020 | PMC7605626 | abstract_body_mismatch | The abstract's ICER values do not match the Results text and Table 2 for either time horizon. |
| Lancet 2012 | PMC10782811 | pct_count | The percentage in the abstract does not match its count. Five states out of the 220 analysed is about 2%, not 11%. An 11% figure would imply a denominator of about 45 states, which is inconsistent with the 220 states use |
| Nat Genet 2021 | PMC7611517 | abstract_body_mismatch | The abstract says no significant effect was observed following maternal BPA exposure, but the Results report a statistically significant increase at IAP-Rnf157 in male offspring exposed to the upper BPA dose. |
| Nat Commun 2026 | PMC13526883 | abstract_body_mismatch | The abstract calls more than 100 channels “usable,” whereas the body distinguishes more than 100 channels observed above the noise floor from 42 channels whose BER meets the stated threshold; it says the total number of |
| JAMA 2025 | PMC12687205 | pct_count | The stated percentages do not equal the numerator/denominator given for either endpoint year. The same 2.92% and 5.18% are repeated in the Discussion. |
| JAMA 2025 | PMC12687205 | pct_count | The reported endpoint percentages do not match their stated numerators and denominators. |
| J Neurosci 2010 | PMC2923025 | p_value | F(1,12)=4.66 corresponds to p around 0.05, not p<0.0001; the reported p-value is impossible for this F statistic. |
| Diabetologia 2017 | PMC6002965 | abstract_body_mismatch | The CI widths for the duration of hypoglycaemia differ drastically between abstract and results; the abstract CIs are implausibly narrow (about ±12 min) compared with the results CIs (about ±300 and ±217 min) for the sam |
| Clin Cancer Res 2008 | PMC2830553 | pct_count | Single-locus instability counts in the newly diagnosed group are described as 4 of 25 samples, but the 3% figure is then attributed to both groups; the abstract reports 3% of patient samples with single-locus MSI, while |
| Clin Cancer Res 2008 | PMC2830553 | abstract_body_mismatch | The abstract reports 3% of patient samples with single-locus MSI, but the results report eight affected samples across the two groups. |
| PLoS Med 2017 | PMC5360219 | claim_vs_data | The discussion reverses the direction of the PHS association described in the results: higher PHS is associated with earlier, not later, expected onset. |
| PLoS Med 2019 | PMC6445413 | abstract_body_mismatch | Abstract says INH and KAN range upper bound 0.73, but Table 3 lists INH fungal ball RPL 3.04 and KAN fungal ball 0.918, and the body text says INH penetration was higher in caseous nodules and fungal ball concentrations |
| Proc Natl Acad Sci U S A 2023 | PMC10691225 | table_arithmetic | Persistence index is defined as a ratio of displacement to path length, which must be ≤1 by definition; reported values of 1.67 and 2.67 are impossible. |
| Proc Natl Acad Sci U S A 2023 | PMC10691225 | impossible_value | The reported persistence-index values are impossible under the paper’s stated definition. |
| Nucleic Acids Res 2008 | PMC2425497 | abstract_body_mismatch | The abstract’s before-calibration accuracy does not agree with the five resolved targets’ Table 4 accuracies weighted by their reported n. The after-calibration accuracy does agree after rounding. |
| Lancet 2012 | PMC3328846 | abstract_body_mismatch | The abstract says practices, patients, pharmacists, and researchers were masked, while the Methods explicitly says those groups were not masked. |
| PLoS Comput Biol 2020 | PMC7332089 | claim_vs_data | The abstract states that no predictor performs better than random on new examples, but the results state that ModPred is better than random on the MS-HeLa dataset, with MCC 0.32. |
| Clin Cancer Res 2010 | PMC3058309 | abstract_body_mismatch | The reported overall prognostic concordance has inconsistent confidence intervals in the abstract, Results, and Table 4, and the Discussion reports a different concordance percentage. |
| J Clin Invest 2025 | PMC12404753 | sample_size | Abstract states 24 adults randomized, but Methods state 26 recruited and 13 randomized per arm; the analysed numbers and exclusions are inconsistent (Results say 2 excluded, Methods say 3 excluded). |
| JAMA Netw Open 2025 | PMC12400126 | abstract_body_mismatch | Obesity F:M median differs between abstract (1.44) and results (2.29) with identical IQR lower/upper bounds. |
| JAMA Netw Open 2025 | PMC12400126 | abstract_body_mismatch | The abstract reports different IQR bounds for the same F:M ratios, and reports a different obesity median from the Results text. |
| Psychol Med 2013 | PMC3936825 | sample_size | The reported overall patient age SD (11.8) is incompatible with the subgroup SDs and ranges in Table 1. The control age SD is also inconsistent: the abstract gives 12.0, while Table 1 gives 6.1 with range 18–42. |
| BMJ 2026 | PMC12915068 | abstract_body_mismatch | The abstract describes 703 as the number of women who underwent bilateral oophorectomy, whereas the Results specify that 703 is the number in that subgroup who died during follow-up. The table corroborates the latter: it |
| Proc Natl Acad Sci U S A 2026 | PMC12956814 | claim_vs_data | The stated instability of the lower-quality-morph fixation boundary is inconsistent with the model’s conformist dynamics at sufficiently high copying probability. In particular, for the stated Type I conformist case (β = |
| Nat Commun 2017 | PMC5458551 | abstract_body_mismatch | The reported wavelength linewidth of 0.642 nm is inconsistent with the stated frequency linewidth of 51 GHz for emission near 737 nm. |
| JAMA Netw Open 2024 | PMC11581535 | claim_vs_data | The claim that both measures decreased or increased with each generation is contradicted by the 1950–1959 to 1960–1969 cohort values. |
| eLife 2025 | PMC11999699 | sample_size | The abstract says samples at all three stages were collected from single individuals, but the methods say each PND8 and PND15 sample pooled testes from two animals. The discussion’s parenthetical qualification indicates |
| Hum Brain Mapp 2011 | PMC3588163 | claim_vs_data | Abstract claims frontal coupling in left middle/inferior frontal gyrus, but the emotion vs neutral coherence results (Table 2 lower panel) list only right inferior/middle frontal gyrus (x=40), not left. |
| Hum Brain Mapp 2011 | PMC3588163 | claim_vs_data | Abstract says frontal coupling is in left middle/inferior frontal gyrus, contradicting Table 2 lower panel which reports only right inferior/middle frontal gyrus for emotion vs neutral. |
| Commun Biol 2025 | PMC12354748 | claim_vs_data | The abstract describes bacterial carbohydrate-metabolism genes as having increased expression, but the paper reports functions predicted from 16S data using Tax4Fun, not gene expression measurements. |
Cost and throughput at scale
| Component | Measured cost | Per 1M papers (on-demand) |
|---|---|---|
| Detectors (CPU) | 0.1 s/paper | ≈ 30 core-hours |
| Screen: Haiku 5.5 low (scale run) | $0.0031/paper | $3,107 |
| Screen: add GPT-6 Luna low (union) | $0.0025/paper | $2,500 |
| Verify: Haiku 5.5 high → GPT-6 Luna high | $0.0011/paper | $1,112 |
| Adjudicate + QC top 0.5% (Opus) | ≈$0.30/candidate | ≈$1,500 |
| Registration track (≈3% of papers are trials) | $0.022/trial | ≈$700 |
| Total | ≈$8.9K (≈ half with batch) |
Throughput on this account's current on-demand quotas: about 6–8 papers/s for Haiku 5.5 and 3–4 for GPT-6 Luna across us./global. profiles, i.e. roughly 1–2 days per million papers. Raising the tokens-per-minute quotas (or Bedrock batch support for these models) is the lever; the code adapts automatically (AIMD gates).
Limitations and validity threats
- Benchmarks: seeded errors are typo-like edits (good for detection, not for severity); real corrections are mostly proof fixes of NIH manuscripts with some noisy ground truth; the found set was discovered by Haiku-based reading and favours it. Matching combines deterministic rules with an LLM fallback (measured ~2.5% false-yes, ~6% false-no on hard cases).
- Oracle precision uses model agreement (Opus + Sol), not human labels; half of sampled flags split the oracle.
- Text only: figures, images and supplementary files are not read; supplements can explain apparent discrepancies.
- Registry checks use the current ClinicalTrials.gov record (history endpoints are not accessible); later registry edits can hide or create discrepancies; protocols and SAPs in supplements were not read.
- Author manuscripts may differ from the version of record; the 10 verified papers are publisher versions.
- Scale-run candidates have two-model agreement only; they need stages 4–5 before anyone is contacted.
Disclosure protocol
- Human expert re-derives the error from the published version (and supplements) and writes a neutral note with quotes and arithmetic.
- Contact the corresponding author privately with a right of reply and time to respond; offer the recomputation.
- If unresolved, contact the journal (COPE guidance); prefer correction requests over public posting.
- Any public note states what is reported, what it should be and why it matters — never speculation about intent or the word "misconduct"; flags are retractable and versioned with the detector/prompt that produced them.