Paper error pipeline: status
Updated 2026-10-28 05:11 UTC. Automated screening of open-access papers in ~140 high-impact journals for internal errors (statistics, numbers, claims vs data). Findings are model-verified candidates, not yet confirmed with authors or journals. Full report: https://contramont.org/published/2026-10-28-KoiqTuLekeHIe4LV2ySPMO/index.html; illustrated write-ups of the strongest verified errors; downloadable lists of confirmed errors: adjudicated_majors.csv, prereg_majors.csv.
Quality vs cost over all experiments
Every pipeline experiment run on the benchmarks is scored the same way, as an end-to-end pipeline, with one quality measure: of the 460 known major errors in the four benchmarks (Seeded errors, PMC-version corrections, Verified real errors, Journal-corrected errors), the share that the pipeline's output reports as a major error. The 45 experiments that ran on all four benchmarks are ranked on one frontier: 18 single screening models, 5 two-model unions, 13 screen + verifier combinations (9 verifiers after the scale-run screen Luna low ∪ Haiku low: Haiku high, Haiku high (materiality rubric), Haiku med, Luna 5.6 high, Luna high, Luna high (materiality rubric), Luna med, Sol med, gpt-oss; and Luna med → Haiku high, Haiku med → Haiku high, Luna med ∪ Haiku low → Haiku high, Luna med ∪ Haiku low → Sol med) and 9 multi-stage pipelines, including the production chain through Opus 5.5 and GPT-6.1 Sol. A flag that both screeners of a union raised is verified once, as in the scale run. Verifiers compared only flag by flag (Opus med, Sonnet med, Terra med) are in the table at the end of this section. Cost is the LLM spend of all stages, averaged per benchmark paper, on one price table. All 184 benchmark × experiment results are in all_experiments.csv, published with the report.
Cost frontier for recall (★): 1 Luna low (generic prompt) ($0.0022, 50.9%, 234/460) · 2 Haiku low (generic prompt) ($0.0033, 54.8%, 252/460) · 3 Haiku low (reconcile prompt) ($0.0044, 58.9%, 271/460) · 4 Luna low ∪ Haiku low ($0.0056, 63.5%, 292/460) · 5 Luna med ∪ Haiku low (generic prompt) ($0.0067, 65.4%, 301/460) · 6 Luna med ∪ Haiku med ($0.0090, 66.3%, 305/460).
On the 225 real major errors alone (seeded errors left out), the cost frontier is close to it: it keeps 5 of its 6 points, drops Haiku low (generic prompt) and adds Luna high, Luna med ∪ Haiku high. Haiku low was run twice with the same settings (250 and 252 of 460 known major errors, $0.0034 and $0.0034 per paper): differences of that size are run-to-run noise, so the repeat run is not ranked.
Recall alone favours raw screeners, which report many unconfirmed flags. The second chart plots precision: the share of output major flags that match a known error (a lower bound, because these papers hold other, unknown errors too). Experiments ringed in both charts are on the three-way frontier: no other experiment is at least as cheap, finds at least as many known major errors and is at least as precise, and strictly better on one of these (23 experiments, ◆ in the table). Many of them win by margins inside sampling error; allowing 2% on recall and precision, 11 remain: Luna low (generic prompt), Luna low, Haiku low (generic prompt), Haiku low, Haiku low (reconcile prompt), Luna med → Haiku high, Luna low ∪ Haiku low, Luna med → Haiku high → Luna high, Luna low ∪ Haiku low → Haiku med, Luna low ∪ Haiku low → Luna med, Luna med ∪ Haiku med.
Cost frontier for precision: 1 Luna low (generic prompt) ($0.0022, 59.2%, 263/444) · 2 Luna low ($0.0022, 63.6%, 246/387) · 3 Luna med → Haiku high ($0.0049, 70.2%, 127/181) · 4 Luna med → Haiku high → Luna high ($0.0057, 74.4%, 119/160) · 5 Luna low ∪ Haiku low → Haiku high → Luna high → Opus high ($0.0382, 76.2%, 141/185) · 6 Luna low ∪ Haiku low → Haiku high → Luna high → Opus high → Sol high ($0.0501, 77.7%, 139/179).
Table: every ranked experiment (45)
| Experiment (shape = kind) | $/paper | Known major errors reported as major [95% CI] (of 460) | …real errors only | All known errors, any severity (of 1120) | Precision (lower bound) | Major flags per clean paper | Failed screen rows | Benchmarks | |
|---|---|---|---|---|---|---|---|---|---|
| ★◆ | Luna low (generic prompt) | $0.0022 | 51% [46%–55%] (234/460) | 44% | 62% | 59% | 0.35 | 1 | all 4 |
| ◆ | Luna low | $0.0022 | 46% [42%–51%] (213/460) | 37% | 62% | 64% | 0.28 | 1 | all 4 |
| Luna low (reconcile prompt) | $0.0028 | 50% [45%–55%] (230/460) | 43% | 60% | 59% | 0.38 | 1 | all 4 | |
| ★◆ | Haiku low (generic prompt) | $0.0033 | 55% [50%–59%] (252/460) | 40% | 57% | 37% | 0.88 | all 4 | |
| Luna med | $0.0033 | 49% [44%–53%] (225/460) | 43% | 67% | 57% | 0.35 | all 4 | ||
| ◆ | Haiku low (detector hints) | $0.0034 | 51% [47%–56%] (235/460) | 39% | 59% | 46% | 0.57 | all 4 | |
| ◆ | Haiku low | $0.0034 | 54% [50%–59%] (250/460) | 40% | 59% | 44% | 0.65 | all 4 | |
| gpt-oss | $0.0034 | 52% [48%–57%] (240/460) | 37% | 41% | 42% | 0.72 | all 4 | ||
| ★◆ | Haiku low (reconcile prompt) | $0.0044 | 59% [54%–63%] (271/460) | 47% | 63% | 45% | 0.66 | all 4 | |
| ◆ | Luna med → Haiku high | $0.0049 | 29% [25%–33%] (132/460) | 25% | 26% | 70% | 0.06 | all 4 | |
| ◆ | Luna high | $0.0052 | 52% [47%–56%] (239/460) | 48% | 72% | 55% | 0.42 | 1 | all 4 |
| ★◆ | Luna low ∪ Haiku low | $0.0056 | 63% [59%–68%] (292/460) | 53% | 73% | 51% | 0.93 | 1 | all 4 |
| Haiku med | $0.0056 | 59% [55%–64%] (273/460) | 48% | 70% | 48% | 0.59 | 16 | all 4 | |
| Luna 5.6 low | $0.0056 | 56% [52%–61%] (259/460) | 44% | 65% | 49% | 0.57 | 2 | all 4 | |
| ◆ | Luna med → Haiku high → Luna high | $0.0057 | 27% [23%–31%] (125/460) | 24% | 13% | 74% | 0.06 | all 4 | |
| ★◆ | Luna med ∪ Haiku low (generic prompt) | $0.0067 | 65% [61%–70%] (301/460) | 56% | 76% | 44% | 1.23 | all 4 | |
| ◆ | Luna med ∪ Haiku low | $0.0067 | 64% [60%–68%] (295/460) | 55% | 76% | 49% | 1.00 | all 4 | |
| Haiku med (reconcile prompt) | $0.0072 | 60% [55%–64%] (275/460) | 52% | 72% | 44% | 0.66 | 90 | all 4 | |
| Haiku med → Haiku high | $0.0078 | 29% [25%–33%] (134/460) | 26% | 31% | 62% | 0.11 | 16 | all 4 | |
| Luna low ∪ Haiku low → gpt-oss | $0.0083 | 19% [16%–23%] (89/460) | 15% | 37% | 72% | 0.05 | 1 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Haiku med | $0.0085 | 32% [28%–37%] (148/460) | 29% | 37% | 68% | 0.12 | 1 | all 4 |
| ◆ | Luna low ∪ Haiku low → Luna med | $0.0088 | 40% [36%–45%] (186/460) | 38% | 35% | 68% | 0.21 | 1 | all 4 |
| Haiku med → Haiku high → Luna high | $0.0088 | 25% [21%–29%] (114/460) | 23% | 13% | 66% | 0.06 | 16 | all 4 | |
| Luna low ∪ Haiku low → Haiku high (materiality rubric) | $0.0089 | 15% [12%–18%] (68/460) | 13% | 37% | 58% | 0.05 | 1 | all 4 | |
| Luna low ∪ Haiku low → Haiku high | $0.0089 | 31% [27%–35%] (143/460) | 27% | 36% | 68% | 0.15 | 1 | all 4 | |
| ★◆ | Luna med ∪ Haiku med | $0.0090 | 66% [62%–70%] (305/460) | 58% | 81% | 51% | 0.94 | 16 | all 4 |
| Luna low ∪ Haiku low → Luna high (materiality rubric) | $0.0101 | 20% [16%–24%] (91/460) | 20% | 37% | 70% | 0.03 | 1 | all 4 | |
| Luna low ∪ Haiku low → Luna high | $0.0101 | 40% [36%–45%] (186/460) | 36% | 36% | 67% | 0.22 | 1 | all 4 | |
| Haiku high | $0.0102 | 60% [55%–64%] (274/460) | 52% | 77% | 45% | 0.62 | 31 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high | $0.0103 | 28% [24%–32%] (128/460) | 24% | 14% | 72% | 0.10 | 1 | all 4 |
| Luna med ∪ Haiku low → Haiku high | $0.0103 | 33% [29%–37%] (152/460) | 31% | 36% | 66% | 0.15 | all 4 | ||
| ◆ | Luna med ∪ Haiku low → Haiku high → Luna high | $0.0117 | 30% [26%–35%] (140/460) | 26% | 15% | 72% | 0.11 | all 4 | |
| Luna low ∪ Haiku low → Haiku high → Luna high (re-grades every flag called real) | $0.0130 | 40% [35%–44%] (183/460) | 35% | 35% | 68% | 0.21 | 1 | all 4 | |
| Luna med ∪ Haiku high | $0.0136 | 66% [61%–70%] (302/460) | 60% | 84% | 50% | 0.97 | 31 | all 4 | |
| ◆ | Luna low ∪ Haiku low → Luna 5.6 high | $0.0190 | 41% [36%–45%] (188/460) | 37% | 36% | 64% | 0.24 | 1 | all 4 |
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high → Opus high | $0.0382 | 24% [20%–28%] (110/460) | 20% | 13% | 76% | 0.04 | 1 | all 4 |
| Sol low | $0.0454 | 52% [47%–56%] (237/460) | 56% | 75% | 45% | – | all 4 | ||
| ◆ | Luna low ∪ Haiku low → Haiku high → Luna high → Opus high → Sol high | $0.0501 | 23% [20%–28%] (108/460) | 20% | 11% | 78% | 0.04 | 1 | all 4 |
| Sonnet low | $0.0518 | 37% [33%–42%] (172/460) | 31% | 64% | 63% | – | all 4 | ||
| Sonnet med | $0.0553 | 38% [34%–42%] (174/460) | 34% | 67% | 61% | – | all 4 | ||
| ◆ | Luna low ∪ Haiku low → Sol med | $0.0600 | 41% [37%–46%] (189/460) | 36% | 36% | 67% | 0.20 | 1 | all 4 |
| Luna low ∪ Haiku low → Luna high → Opus high | $0.0606 | 27% [23%–31%] (122/460) | 22% | 20% | 74% | 0.06 | 1 | all 4 | |
| Luna med ∪ Haiku low → Sol med | $0.0649 | 41% [37%–46%] (190/460) | 37% | 35% | 63% | 0.27 | all 4 | ||
| ◆ | Luna low ∪ Haiku low → Luna high → Opus high → Sol high | $0.0750 | 26% [22%–30%] (118/460) | 21% | 12% | 75% | 0.05 | 1 | all 4 |
| ◆ | Opus med | $0.1727 | 46% [42%–51%] (212/460) | 42% | 81% | 65% | – | 2 | all 4 |
★ cost frontier for major-error recall · ◆ three-way frontier. Pooled Wilson intervals treat errors as independent. Major flags per clean paper is measured on the 240 unmodified control papers of the seeded benchmark (flags there include real errors in the originals, so it is an upper bound on false alarms; – where controls were not run: Opus, Sol and Sonnet screened the 240 papers with an injected error only). Papers whose screener output could not be used count as finding nothing, and unions and cascades built on that screener inherit them (column 'Failed screen rows'): Haiku med (reconcile prompt) 90 of 1002, Haiku high 31 of 1002, Haiku med 16 of 1002, Luna 5.6 low 2 of 1002, Opus med 2 of 762, Luna low (generic prompt) 1 of 1002. Most were cut off at the 16,000-token output cap; GPT-6 Luna rows that failed only because of throttling were re-run, and Opus 5.5 refusals were re-asked of Opus 5.
Not ranked: repeat runs and experiments not run on all four benchmarks (1)
| Experiment (shape = kind) | $/paper | Known major errors reported as major [95% CI] (of 460) | …real errors only | All known errors, any severity (of 1120) | Precision (lower bound) | Major flags per clean paper | Failed screen rows | Benchmarks | |
|---|---|---|---|---|---|---|---|---|---|
| Haiku low (repeat run) | $0.0034 | 55% [50%–59%] (252/460) | 43% | 61% | 44% | 0.64 | 1 | all 4 |
Pooled over the benchmarks each ran on; rows on fewer benchmarks are not comparable with the ranked rows.
Verifiers judged flag by flag
A second view of the verifiers above (except Luna med, not run on this flag pool): agreement with the Opus 5.5 + GPT-6.1 Sol oracle on the found-set flag pool; ★ marks the cost frontier for agreement on real vs not (acc) and for recall of oracle-major flags. The two Oracle rows are the oracle runs themselves (100% agreement by construction in the first three columns; the last column is scored against the known errors).
| Verifier | $/flag | Agrees with oracle (real?) | Precision on major | Recall on major | Keeps known majors major |
|---|---|---|---|---|---|
| ★ Haiku high (materiality rubric) | $0.0029 | 88% ★ | 95% | 44% ★ | 50% |
| ★ Haiku med | $0.0030 | 86% | 96% | 61% ★ | 58% |
| gpt-oss | $0.0033 | 85% | 86% | 44% | 46% |
| ★ Haiku high | $0.0034 | 88% | 97% | 71% ★ | 69% |
| ★ Luna high (materiality rubric) | $0.0040 | 94% ★ | 100% | 49% | 54% |
| ★ Luna high | $0.0045 | 95% ★ | 93% | 93% ★ | 85% |
| ★ Luna 5.6 high | $0.0119 | 94% | 85% | 95% ★ | 88% |
| ★ Terra med | $0.0437 | 95% ★ | 88% | 85% | 77% |
| Sonnet med | $0.0501 | 90% | 100% | 17% | 15% |
| ★ Sol med | $0.0549 | 99% ★ | 95% | 98% ★ | 88% |
| Oracle: Sol high | $0.0688 | 100% | 100% | 100% | 88% |
| Opus med | $0.1068 | 98% | 100% | 90% | 85% |
| Oracle: Opus high | $0.1138 | 100% | 100% | 100% | 73% |
Not on this frontier, because they ran on scale-run candidates, which have no ground truth: the order in which candidates are adjudicated (learned ranker vs baseline, A/B), the adversarial defence of adjudicated majors (GPT-6.1 Sol defends, Opus 5.5 rules), the Opus 5 fallback for Opus 5.5 refusals, and the materiality rubric at scale (its benchmark versions, the materiality-rubric verifiers, are ranked above). Per-benchmark screener frontiers and the stage-by-stage view on journal-corrected errors follow below.
Screeners, benchmark by benchmark
Seeded errors (240 papers with one injected error; 240 clean controls)
Frontier: 1 Luna low (generic) ($0.0021, 91%) · 2 Luna low ($0.0022, 92%) · 3 Luna med ($0.0033, 95%) · 4 Luna high ($0.0052, 98%) · 5 Luna med ∪ Haiku low (generic) ($0.0067, 99%) · 6 Luna med ∪ Haiku med ($0.0091, 100%)
Recall = injected errors flagged at any severity (LLM-assisted matching). Expensive models (Sol, Sonnet, Opus) were run on the mutated papers only, so they have no control false-alarm rate.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0021 | 91% [88–95] | 57% | 0.35 |
| ★ luna6-low-checklist | OpenAI | $0.0022 | 92% [88–95] | 54% | 0.28 |
| luna6-low-reconcile | OpenAI | $0.0028 | 92% [88–95] | 56% | 0.38 |
| ★ luna6-med-checklist | OpenAI | $0.0033 | 95% [92–98] | 53% | 0.35 |
| haiku55-low-generic | Claude | $0.0034 | 90% [86–94] | 68% | 0.88 |
| haiku55-low-hints | Claude | $0.0034 | 93% [89–96] | 62% | 0.57 |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 88% [84–92] | 65% | 0.64 |
| haiku55-low-checklist | Claude | $0.0035 | 88% [84–92] | 67% | 0.65 |
| gptoss120-checklist | OpenAI (open-weight) | $0.0035 | 75% [69–80] | 66% | 0.72 |
| haiku55-low-reconcile | Claude | $0.0046 | 92% [89–96] | 69% | 0.66 |
| ★ luna6-high-checklist | OpenAI | $0.0052 | 98% [96–100] | 54% | 0.42 |
| luna56-low-checklist | OpenAI | $0.0056 | 90% [86–94] | 66% | 0.57 |
| luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0057 | 96% [93–98] | 72% | 0.93 |
| haiku55-med-checklist | Claude | $0.0058 | 97% [95–99] | 69% | 0.59 |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0067 | 99% [98–100] | 73% | 1.23 |
| luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0068 | 97% [95–99] | 71% | 1.00 |
| haiku55-med-reconcile | Claude | $0.0075 | 90% [87–94] | 65% | 0.66 |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0091 | 100% [100–100] | 73% | 0.94 |
| haiku55-high-checklist | Claude | $0.0106 | 97% [95–99] | 65% | 0.62 |
| luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0139 | 100% [99–100] | 70% | 0.97 |
| sol61-low-checklist | OpenAI | $0.0457 | 98% [97–100] | 46% | |
| sonnet55-low-checklist | Claude | $0.0510 | 96% [93–98] | 42% | |
| sonnet55-med-checklist | Claude | $0.0552 | 95% [92–98] | 40% | |
| opus55-med-checklist | Claude | $0.1590 | 99% [97–100] | 50% |
Real corrections detectable from the original text (113 papers, 205 corrected values)
Frontier: 1 Luna low (generic) ($0.0023, 54%) · 2 Luna med ($0.0036, 57%) · 3 Luna high ($0.0056, 60%) · 4 Haiku med ($0.0058, 62%) · 5 Luna low ∪ Haiku low ($0.0058, 66%) · 6 Luna med ∪ Haiku low ($0.0070, 68%) · 7 Luna med ∪ Haiku low (generic) ($0.0070, 68%) · 8 Luna med ∪ Haiku med ($0.0093, 73%) · 9 Luna med ∪ Haiku high ($0.0142, 75%)
Values that a later version of the paper corrected and that an attentive reader could have spotted in the original. Recall at any severity; most corrections are minor, so 'flagged major' is low by design.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0023 | 54% [43–63] | 10% | |
| luna6-low-checklist | OpenAI | $0.0024 | 52% [41–61] | 9% | |
| luna6-low-reconcile | OpenAI | $0.0028 | 45% [34–55] | 11% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0034 | 36% [23–48] | 20% | |
| haiku55-low-checklist | Claude | $0.0034 | 48% [37–58] | 5% | |
| haiku55-low-hints | Claude | $0.0034 | 49% [38–59] | 4% | |
| haiku55-low-generic | Claude | $0.0035 | 51% [42–58] | 11% | |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 52% [41–61] | 5% | |
| ★ luna6-med-checklist | OpenAI | $0.0036 | 57% [47–66] | 11% | |
| haiku55-low-reconcile | Claude | $0.0045 | 47% [38–56] | 11% | |
| ★ luna6-high-checklist | OpenAI | $0.0056 | 60% [51–69] | 6% | |
| ★ haiku55-med-checklist | Claude | $0.0058 | 62% [53–71] | 8% | |
| ★ luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0058 | 66% [56–74] | 13% | |
| luna56-low-checklist | OpenAI | $0.0060 | 60% [49–68] | 20% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0070 | 68% [59–75] | 13% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0070 | 68% [59–75] | 17% | |
| haiku55-med-reconcile | Claude | $0.0075 | 65% [56–74] | 7% | |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0093 | 73% [64–81] | 15% | |
| haiku55-high-checklist | Claude | $0.0106 | 68% [56–77] | 8% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0142 | 75% [66–83] | 15% | |
| sol61-low-checklist | OpenAI | $0.0477 | 62% [53–71] | 15% | |
| sonnet55-low-checklist | Claude | $0.0573 | 52% [43–61] | 3% | |
| sonnet55-med-checklist | Claude | $0.0593 | 60% [51–69] | 1% | |
| opus55-med-checklist | Claude | $0.2165 | 73% [63–81] | 10% |
Journal-corrected errors detectable from the original text (381 papers, 647 corrected values)
Frontier: 1 Luna low (generic) ($0.0021, 54%) · 2 Luna low ($0.0022, 55%) · 3 Luna med ($0.0033, 60%) · 4 Luna high ($0.0050, 67%) · 5 Luna med ∪ Haiku low (generic) ($0.0065, 70%) · 6 Luna med ∪ Haiku low ($0.0065, 70%) · 7 Luna med ∪ Haiku med ($0.0086, 75%) · 8 Luna med ∪ Haiku high ($0.0129, 80%)
Errors a journal later corrected in a published notice, still present in the archived original, and judged detectable from the paper alone by two models. Recall at any severity, LLM-assisted matching.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0021 | 54% [49–59] | 23% | |
| ★ luna6-low-checklist | OpenAI | $0.0022 | 55% [49–60] | 20% | |
| luna6-low-reconcile | OpenAI | $0.0027 | 52% [47–57] | 19% | |
| haiku55-low-generic | Claude | $0.0032 | 46% [41–52] | 22% | |
| haiku55-low-checklist | Claude | $0.0032 | 50% [45–56] | 23% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0032 | 30% [25–35] | 22% | |
| haiku55-low-checklist-s1 | Claude | $0.0032 | 53% [48–58] | 24% | |
| haiku55-low-hints | Claude | $0.0032 | 49% [44–54] | 20% | |
| ★ luna6-med-checklist | OpenAI | $0.0033 | 60% [56–65] | 19% | |
| haiku55-low-reconcile | Claude | $0.0041 | 57% [52–62] | 26% | |
| ★ luna6-high-checklist | OpenAI | $0.0050 | 67% [62–71] | 26% | |
| haiku55-med-checklist | Claude | $0.0053 | 62% [58–66] | 23% | |
| luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0054 | 67% [62–71] | 32% | |
| luna56-low-checklist | OpenAI | $0.0055 | 57% [52–63] | 25% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0065 | 70% [66–74] | 30% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0065 | 70% [66–75] | 30% | |
| haiku55-med-reconcile | Claude | $0.0067 | 66% [61–72] | 28% | |
| ★ luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0086 | 75% [72–79] | 30% | |
| haiku55-high-checklist | Claude | $0.0096 | 72% [67–76] | 24% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0129 | 80% [77–84] | 30% | |
| sol61-low-checklist | OpenAI | $0.0442 | 70% [66–75] | 29% | |
| sonnet55-low-checklist | Claude | $0.0506 | 55% [51–60] | 14% | |
| sonnet55-med-checklist | Claude | $0.0540 | 57% [53–62] | 14% | |
| opus55-med-checklist | Claude | $0.1684 | 77% [73–81] | 17% |
Independently verified real errors (28 papers: 19 major, 9 minor errors)
Frontier: 1 Luna low (generic) ($0.0023, 61%) · 2 Luna low (reconcile) ($0.0031, 64%) · 3 Haiku low (generic) ($0.0034, 86%) · 4 Luna low ∪ Haiku low ($0.0059, 89%) · 5 Luna med ∪ Haiku low (generic) ($0.0071, 93%) · 6 Luna med ∪ Haiku low ($0.0073, 96%) · 7 Luna med ∪ Haiku high ($0.0149, 100%)
Real errors confirmed earlier in published papers. Recall at any severity, LLM-assisted matching. The set was first found by Haiku-based reading, so it favours Haiku; it has no clean controls.
Table view
| Config | Family | $/paper | Recall [95% CI] | Recall, flagged major | Major false alarms per clean paper |
|---|---|---|---|---|---|
| ★ luna6-low-generic | OpenAI | $0.0023 | 61% [43–79] | 57% | |
| luna6-low-checklist | OpenAI | $0.0023 | 61% [43–79] | 57% | |
| ★ luna6-low-reconcile | OpenAI | $0.0031 | 64% [46–82] | 54% | |
| ★ haiku55-low-generic | Claude | $0.0034 | 86% [71–96] | 82% | |
| haiku55-low-hints | Claude | $0.0034 | 79% [61–93] | 75% | |
| haiku55-low-checklist-s1 | Claude | $0.0035 | 79% [64–93] | 68% | |
| haiku55-low-checklist | Claude | $0.0036 | 79% [61–93] | 75% | |
| gptoss120-checklist | OpenAI (open-weight) | $0.0036 | 43% [25–61] | 32% | |
| luna6-med-checklist | OpenAI | $0.0037 | 79% [64–93] | 75% | |
| haiku55-low-reconcile | Claude | $0.0049 | 86% [73–96] | 79% | |
| ★ luna6-low-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0059 | 89% [79–100] | 86% | |
| luna56-low-checklist | OpenAI | $0.0060 | 57% [39–75] | 57% | |
| haiku55-med-checklist | Claude | $0.0061 | 86% [71–96] | 79% | |
| luna6-high-checklist | OpenAI | $0.0062 | 75% [61–89] | 75% | |
| ★ luna6-med-checklist ∪ haiku55-low-generic | Ensemble (Claude ∪ OpenAI) | $0.0071 | 93% [82–100] | 93% | |
| ★ luna6-med-checklist ∪ haiku55-low-checklist | Ensemble (Claude ∪ OpenAI) | $0.0073 | 96% [89–100] | 93% | |
| haiku55-med-reconcile | Claude | $0.0080 | 86% [71–96] | 82% | |
| luna6-med-checklist ∪ haiku55-med-checklist | Ensemble (Claude ∪ OpenAI) | $0.0098 | 96% [89–100] | 89% | |
| haiku55-high-checklist | Claude | $0.0113 | 93% [82–100] | 86% | |
| ★ luna6-med-checklist ∪ haiku55-high-checklist | Ensemble (Claude ∪ OpenAI) | $0.0149 | 100% [100–100] | 96% | |
| sol61-low-checklist | OpenAI | $0.0496 | 89% [75–100] | 89% | |
| sonnet55-low-checklist | Claude | $0.0530 | 79% [64–93] | 39% | |
| sonnet55-med-checklist | Claude | $0.0590 | 82% [68–96] | 57% | |
| opus55-med-checklist | Claude | $0.1738 | 93% [82–100] | 68% |
End-to-end pipeline frontier on journal-corrected errors
| Stage | Cumulative $/paper | Major corrected errors still held as major (95% CI) | All corrected errors | Flags/paper passed on | Precision of this stage's output (measured on the scale run) |
|---|---|---|---|---|---|
| Screen: Luna-low ∪ Haiku-low (any flag) | $0.0054 | 74% [66%–81%] (126/170) | 67% | 6.96 | |
| Screen: flagged major | $0.0054 | 57% [48%–66%] (97/170) | 32% | 1.16 | |
| + Haiku 5.5 high verifies | $0.0084 | 29% [19%–38%] (49/170) | 11% | 0.28 | |
| + GPT-6 Luna high verifies (candidates) | $0.0095 | 25% [16%–35%] (42/170) | 9% | 0.21 | 43% both-major, 79% real (random scale sample, Opus+Sol) |
| + Opus 5.5 high adjudicates | $0.0306 | 21% [12%–31%] (35/170) | 7% | 0.15 | |
| + GPT-6.1 Sol high adjudicates (final list) | $0.0392 | 20% [11%–30%] (34/170) | 7% | 0.14 | 20/20 real, 13/20 major and uncorrected (agent audit); 33/40 upheld under adversarial review |
| Variant: Haiku-high only removes non-errors; Luna-high makes the severity call | $0.0122 | 36% [28%–47%] (62/170) | 16% | 0.41 | |
| Variant: Luna-high is the only cheap verifier | $0.0096 | 38% [29%–48%] (64/170) | 16% | 0.42 | |
| Variant: Luna-high only, then Opus + Sol adjudicate (final list) | $0.0667 | 22% [13%–32%] (37/170) | 8% | 0.18 | |
| Alternative: GPT-6.1 Sol medium is the only verifier | $0.0536 | 39% [29%–50%] (67/170) | 15% | 0.44 |
381 papers whose errors the journals later corrected, restricted to errors a careful reader could have spotted in the original (both Claude Haiku 5.5 and GPT-6 Luna agree). Each verification stage passes on only the flags it rates major, so recall can only fall while precision rises; costs are per benchmark paper, which carry more flags than an average paper. The first row counts a corrected error as found if any screener flag points at it.
Verifier experiment (second tier)
Pool: 244 screener flags. Oracle labels: 184 real (41 major), 36 not real, 24 oracle disagreements.
| Verifier | $/flag | Agrees with oracle (real?) | Precision on major | Recall on major | Confirms known errors | Keeps known majors major |
|---|---|---|---|---|---|---|
| haiku55-high-material | $0.0029 | 88% | 95% | 44% | 95% | 50% |
| haiku55-med | $0.0030 | 86% | 96% | 61% | 95% | 58% |
| gptoss120 | $0.0033 | 85% | 86% | 44% | 98% | 46% |
| haiku55-high | $0.0034 | 88% | 97% | 71% | 98% | 69% |
| luna6-high-material | $0.0040 | 94% | 100% | 49% | 93% | 54% |
| luna6-high | $0.0045 | 95% | 93% | 93% | 93% | 85% |
| luna56-high | $0.0119 | 94% | 85% | 95% | 93% | 88% |
| terra56-med | $0.0437 | 95% | 88% | 85% | 90% | 77% |
| sonnet55-med | $0.0501 | 90% | 100% | 17% | 98% | 15% |
| sol61-med | $0.0549 | 99% | 95% | 98% | 95% | 88% |
| oracle-sol61-high | $0.0688 | 100% | 100% | 100% | 95% | 88% |
| opus55-med | $0.1068 | 98% | 100% | 90% | 93% | 85% |
| oracle-opus55-high | $0.1138 | 100% | 100% | 100% | 93% | 73% |
Each verifier sees the full paper plus one flagged issue and decides whether it is a real error and whether it is major. Oracle = agreement of Opus 5.5 (high effort) and GPT-6.1 Sol (high effort); rows named oracle-* are those two runs, and GPT-6.1 Sol at medium effort is correlated with its own oracle half, so treat its agreement as an upper bound. "Confirms known errors" = share of flags matching one of the 28 known errors that the verifier accepts.
Severity-rubric experiment. The default rubric calls any error in the abstract or a headline result major; a 'materiality' rubric (major only if correcting it changes a conclusion, significance or a headline estimate materially) was tried on the cheap verifiers to see whether they then agree better with Opus+Sol. It does not: precision against Opus+Sol barely moves while half the confirmed majors are dropped, and on the found set it keeps far fewer known major errors (rows ending in -material above). The default rubric stays; strong-model adjudication of top-ranked candidates does the severity call.
| Extra filter on two-model candidates | Kept | Opus+Sol both major | Share of Opus+Sol majors kept |
|---|---|---|---|
| No extra filter (random candidates) | 120/120 | 43% | 100% |
| haiku55-high-material (random candidates) | 53/120 | 53% | 54% |
| luna6-high-material (random candidates) | 58/120 | 57% | 63% |
| haiku55-high-material & luna6-high-material (random candidates) | 36/120 | 61% | 42% |
| No extra filter (top-ranked candidates) | 200/200 | 68% | 100% |
| haiku55-high-material (top-ranked candidates) | 102/200 | 75% | 56% |
| luna6-high-material (top-ranked candidates) | 98/200 | 68% | 49% |
| haiku55-high-material & luna6-high-material (top-ranked candidates) | 68/200 | 74% | 37% |
End-to-end cascade
| Stage | $/paper, all stages | Known major errors still reported as major | …of them real (not seeded) | Precision (lower bound) |
|---|---|---|---|---|
| Screen: Luna low ∪ Haiku low (the scale-run screen) | $0.0056 | 292/460 (63%) | 120/225 | 51% |
| + Claude Haiku 5.5 high verifies | $0.0089 | 143/460 (31%) | 61/225 | 68% |
| + GPT-6 Luna high verifies (both must say major): the candidates | $0.0103 | 128/460 (28%) | 53/225 | 72% |
| + Opus 5.5 high adjudicates | $0.0382 | 110/460 (24%) | 45/225 | 76% |
| + GPT-6.1 Sol high adjudicates: the final list | $0.0501 | 108/460 (23%) | 44/225 | 78% |
| Alternative: GPT-6 Luna high as the only verifier | $0.0101 | 186/460 (40%) | 81/225 | 67% |
| Alternative: GPT-6.1 Sol medium as the only verifier | $0.0600 | 189/460 (41%) | 82/225 | 67% |
Pooled over the four benchmarks (seeded, PMC-version corrections, journal-corrected, verified real errors). Requiring both cheap verifiers to agree raises precision (72%, against 67% for Luna high alone and 67% for Sol medium) but keeps fewer known major errors: 128 of 460, against 186 for Luna high alone ($0.0101 per paper) and 189 for Sol medium ($0.0600). At scale the pair's job is to cut the stream of flags cheaply before the expensive judges.
On a uniform random sample of 120 scale-run candidates, Opus 5.5 (high) and GPT-6.1 Sol (high) both called 43% a real major error (95% CI 35%–52%) and 79% a real error of any severity (71%–85%). The judges differ mainly on severity: Sol alone calls 76% major, Opus alone 50%.
Pilot on 2,000 random high-impact papers
| Stage | Count | Cost |
|---|---|---|
| Papers screened (Haiku-low ∪ Luna-med) | 2,000 | $13.47 |
| Flags (all severities) | 10,446 | |
| Major flags (confidence ≥ 0.5) | 695 | |
| Luna-high verifier: confirmed major | 259 | $3.24 |
| GPT-6.1 Sol judge: confirmed major | 191 in 140 papers | $14.76 |
Total $31.47 = $0.0157 per paper.
Realistic precision on random papers (oracle = Opus 5.5 high and GPT-6.1 Sol high agreeing, 150 sampled flags, half decided by the oracle): Luna-high alone 0.56 · Luna-high → GPT-6.1 Sol judge 0.92 · Luna-high → Haiku-high (both say major) 0.97 at 6% of the judge's cost. Luna rejections lost 1 of 35 sampled majors. These figures count only flags on which the two oracle models agreed about severity, which overstates precision; the scale-run section gives an unbiased random-sample estimate.
Scale run: all 438,528 research articles in scope
| Screener | Papers screened | Major flags | Cost |
|---|---|---|---|
| haiku55-low-checklist | 438,528 | 124,881 | $1362.39 |
| luna6-low-checklist | 265,000 | 33,759 | $554.50 |
| luna6-med-checklist | 5,000 | 836 | $16.17 |
Cascade (final): 123,861 major flags → Haiku-high confirmed 21,111 → Luna-high confirmed 13,497 candidates (verification cost $487.43).
On-demand Bedrock (batch is not offered for these models). Complete as of 2026-10-23 15:50 UTC: Claude Haiku 5.5 read every paper in scope; GPT-6 Luna, limited by its throughput quota, read the first 260,000 in the same order as a second screener before the run reached its $4,500 cap (so those papers had both).
Adjudicated: 5726 major errors in 4482 papers
Top 9400 scale-run candidates re-checked independently by Opus 5.5 (high) and GPT-6.1 Sol (high; from 18:30 UTC only on what Opus calls major, since Opus is the stricter judge); 5726 (61%) were called a real major error by both. Not yet hand-verified. Opus 5.5 refused about 10% of these requests (stop reason 'refusal' on ordinary papers); 1158 were judged by Opus 5 (high) instead. On 50 candidates both answered, Opus 5 and Opus 5.5 agree on real vs not in 90% and on major vs not in 72%, with similar major rates (26 vs 28). Adversarial check (GPT-6.1 Sol builds the strongest honest defense of the authors, Opus rules) on a random 40 adjudicated majors: 33 (82%, 95% CI 68%–91%) upheld as major, 7 downgraded to minor, 0 judged not an error; on a random 40 from the most recent, lowest-ranked 2,000 adjudications: 30 (75%, 95% CI 60%–86%) upheld as major, 10 downgraded to minor, 0 judged not an error. Downgrades are mostly a slip whose correct value is unambiguous elsewhere in the paper. Independent agent audit (Claude agents with Python and web access; one reproduces the error from scratch, one argues the authors' side, one judges materiality and searches for published corrections) on a random 20: 20/20 reproduced as real errors, 13/20 (65%, 95% CI 43%–82%) also judged major and uncorrected; on the 20 highest-confidence: 20/20 reproduced as real errors, 18/20 (90%, 95% CI 70%–97%) also judged major and uncorrected, 1 already corrected by the journal. Published notices: of 4,084 flagged papers swept (PMC, Crossref, Europe PMC), 273 have a correction, retraction or expression of concern. For 42 confirmed findings the notice fixes the very error we flagged from the original text (both Haiku 5.5 and GPT-6 Luna agree) - independent confirmation by the journals themselves; those rows are marked as already corrected in the downloads. 142 notices concern something else, and 158 are too thin to tell. Author manuscripts: 1,639 of 5,726 confirmed findings (29%) are in papers whose PMC copy is the authors' accepted manuscript rather than the journal's final version. Errors can be fixed during copy-editing: in the write-up checks, 5 of 18 such errors were absent from the publisher's version of record, so roughly 8% of the list may not appear in the final published paper. The downloads flag these rows. Ordering experiment: since 22:30 UTC adjudication batches are split between a learned ranker (logistic regression on the cheap verifiers' confidences, flag category and text, retrained on all adjudications so far; before the experiment, 5-fold cross-validated AUC within the 2,950 top-ranked candidates adjudicated by then was 0.71 vs 0.61 for mean confidence, and on 66 random-sample candidates outside that set both orderings scored about 0.50, too few to separate them) and the cheap verifiers' mean confidence, half and half for the first batch, 85/15 for the next five, and all learned from 05:25 UTC. Both-major so far: baseline 263/645 (41%), learned 3327/5505 (60%) (Fisher exact p = 2e-21).
For comparison: On a uniform random sample of 120 scale-run candidates, Opus 5.5 (high) and GPT-6.1 Sol (high) both called 43% a real major error (95% CI 35%–52%) and 79% a real error of any severity (71%–85%). The judges differ mainly on severity: Sol alone calls 76% major, Opus alone 50%. Taking the top-ranked candidates first raises the both-major rate from 43% to 61%.
List
| Journal | PMCID | Category | Opus summary |
|---|---|---|---|
| BMC Med 2025 | PMC12211446 | ci | The confidence intervals for the headline prevalence in the abstract and results, and for every genotype-specific prevalence, are mathematically impossible for n = 822. The correct 95% CI for 6.2% is roughly 4.6%-7.9%, not 6.18%-6.22%. Because this sits in the abstract's headline result and greatly |
| Nat Commun 2025 | PMC12575868 | table_arithmetic | The nationwide savings figure is overstated by a factor of 1000. The stated inputs (26.72 MJ/m² × 97 billion ft²) give about 66.9 million MWh, not 66.87 billion MWh. The 66.87 billion MWh figure is also physically impossible, being far more than total U.S. electricity use. The same factor-of-1000 er |
| Nat Commun 2017 | PMC5309857 | p_value | Genuine error. The Fisher exact and χ² p-values in Table 1 cannot come from the tabulated mouse counts: recomputation gives two-sided p≈0.26 (reported 0.0006), p≈0.08 (reported <0.0001) and 3×2 χ² p≈0.15 (reported <0.0001). The reported values are exactly what one obtains by running Fisher's test on |
| Ann Rheum Dis 2011 | PMC3232181 | ci | Confirmed genuine error: the meta-analysis result for FCGR2A and malar rash is reported as OR 1.11 with 95% CI 1.17 to 1.33, an interval that excludes its own point estimate and is therefore mathematically impossible. Back-calculation from the reported p=0.0031 gives a plausible CI of about 1.04-1.1 |
| JAMA Netw Open 2025 | PMC12457976 | abstract_body_mismatch | The abstract attributes the overall-cohort HLOS estimates to the hypotension subgroup. The actual subgroup estimates are SAE -1.65 (P=.04) and observation -0.13 (P=.86, not significant). The abstract and conclusions therefore falsely claim significantly shorter LOS with observation in hypotensive pa |
| Clin Cancer Res 2012 | PMC3596113 | p_value | The reported Fisher exact P = 0.0005 for 1/6 vs 7/18 CR/CRi is impossible. The correct two-sided Fisher P is about 0.62 (one-sided about 0.32). The error turns a non-significant difference into an apparently highly significant one. The authors use it as statistical support for the dose-response clai |
| JAMA Netw Open 2020 | PMC8094425 | p_value | The bone-only metastasis OS subgroup is reported as HR 1.22 (95% CI 0.88-1.68) with P < .001. The CI includes 1, and the recomputed two-sided P is about 0.23. The result therefore is not statistically significant. This error appears in the abstract, and it directly supports the paper's stated conclu |
| JAMA Netw Open 2025 | PMC12406064 | p_value | The reported P < .001 cannot follow from the reported t129 = 0.71 (P ≈ .48) or from the heavily overlapping CIs. The between-group difference in HDRS-17 change is not statistically significant. Because the abstract and the Results present this secondary outcome as 'significantly' greater with MBC, t |
| Sci Adv 2023 | PMC9891701 | claim_vs_data | Genuine error confirmed. The abstract ('Depletion of pancreatic-specific TASc promoted the tumorigenesis of PDAC tumors') and the Introduction ('Depletion of TASc promoted the tumorigenesis of PDAC') state the opposite direction of the paper's own findings: AAV8-GFAP-DTA–mediated TASc depletion impr |
| Cell Rep 2026 | PMC13560472 | abstract_body_mismatch | Confirmed. The Summary abstract and the 'In brief' both state that high metformin concentrations 'upregulate G6PC expression ... to prevent glucose release' in IECs, whereas the Results, Discussion, Figure 6 title, and the paper's closing summary all report that metformin significantly DECREASES/dow |
| JAMA Netw Open 2020 | PMC8094425 | p_value | The bone-only metastasis OS subgroup has HR 1.22 with a 95% CI of 0.88-1.68, which includes 1 and implies p of about 0.23. Yet the paper reports P < .001 in both the abstract and the results. The Discussion and Conclusions also explicitly claim significantly prolonged OS in this subgroup. This is a |
| BMC Med 2025 | PMC12403310 | claim_vs_data | The paper's own Results, Table 2, Table 3 and Discussion show 216/376 (57.4%) trials were concordant and 160/376 (42.6%) discordant. The abstract and both Conclusions sections say 'over half (216/376, 57.4%) had discordance', which mislabels the concordant group and turns the headline finding around |
| JAMA 2010 | PMC3655326 | p_value | Fisher's exact test on the reported renal cancer counts (4/45 vs 6/155) gives a two-sided p of about 0.24, not 0.004. The renal cancer association is therefore not statistically significant. It is a headline claim in the abstract and conclusions and is used to recommend renal surveillance, so the er |
| Clin Cancer Res 2012 | PMC3596113 | claim_vs_data | The Fisher exact P for 1/6 vs 7/18 is about 0.62 two-sided (0.32 one-sided), not 0.0005. The reported value turns a non-significant difference into a highly significant one. The Discussion uses it as evidence of a dose-response relationship supporting SCH 900776's contribution, so this is a genuine |
| Hum Brain Mapp 2022 | PMC9057088 | table_arithmetic | Table 4's T and p columns contradict each other and the reported means and SDs. - Large t values (8.6–12.9) are paired with p-values up to .995, which is impossible. - Recomputed t values for temporal, TH and whole brain are all below 1. - The abstract's claim that frontal volume is lower in ADHD (r |
| JAMA Netw Open 2026 | PMC13434869 | ci | The risk-difference CIs for the primary outcome, reported in both the abstract and the Results, are grossly too narrow. Recomputed from Table 2 counts, the correct 95% CIs are about 0.8–8.5, 5.7–13.0 and 1.2–8.1 pp. The paper's own z statistics confirm SEs of about 1.8–2.0 pp. The NNT CIs share the |
| Nat Commun 2025 | PMC12575868 | table_arithmetic | The stated inputs (9.01e9 m² × 26.72 MJ/m²) give about 66.9 million MWh, not 66.87 billion MWh. This is a 1000-fold unit error. It propagates to the $7.35 trillion cost and 66.94 billion t CO2 figures, all of which appear in the abstract and discussion as headline results, and the claimed total far |
| Nat Commun 2017 | PMC5309857 | p_value | Genuine error. The Fisher exact p-values in Table 1 cannot come from the tabulated mouse counts: 9/16 vs 4/13 positive gives two-tailed Fisher p = 0.26 (chi-square p = 0.17), not 0.0006; 4/13 vs 13/20 gives p ~0.08, not <0.0001; and the 3x2 chi-square gives p = 0.15, not <0.0001. The reported values |
| J Hepatol 2020 | PMC7395653 | ci | The reported 95% CIs are implausibly narrow and inconsistent with the stated Poisson assumption, and this applies to both the SIR and the SMR. Correctly computed, the SIR CI is about 157–230: the result stays significant, but the abstract's interval is wrong. The SMR CI with 20 deaths is about 58–14 |
| Lancet Reg Health Eur 2025 | PMC11910110 | abstract_body_mismatch | The error is genuine. The abstract's sentence on preterm birth reports low-birth-weight estimates from Table 4 for ADHD (4·15), ASD (3·23) and non-verbal IQ (-5·44). For ASD this turns a null preterm association (0·60, CI -2·60 to 3·79) into a stated significant one, contradicting the Results ('Not |
| Lancet HIV 2021 | PMC7869891 | ci | The Wilson upper 95% bound for 0/384 DTG-exposed outcomes is about 0.0099, not the reported 0.0010. The EFV bound (0.0036 for 0/1068) checks out under the same method, which confirms that the DTG value is wrong. This is the headline NTD estimate and it appears in the abstract. The wrong value unders |
| BMC Med 2023 | PMC10642059 | claim_vs_data | The abstract says all PPD-related variables were significantly associated with CVD mortality. Table 2 and the Results text show the opposite: no PPD measure was significantly associated with CVD mortality (P=0.275, 0.302, 0.337). This is a direct reversal of a result stated in the abstract, so it is |
| BMJ Glob Health 2025 | PMC11749663 | claim_vs_data | The RR of 2.55 (0.82 to 4.27) for complete abortion is mathematically impossible. The control risk is 96.1%, and Table 2's own figures (96.1% vs 97.3%, RD 1.49%) imply an RR of about 1.01–1.02. The CI is also symmetric on the arithmetic scale, which is inconsistent with a ratio. The erroneous value |
| Diabetes Care 2009 | PMC2732170 | table_arithmetic | The VA extrapolated costs in Table 3 were computed with the 75% population share instead of the stated 5%; the MCLNO and TUHC rows reproduce exactly with their stated shares. This inflates the VA cost from about $17M to $260M. The headline population lifetime cost in the abstract is therefore about |
| BMC Med 2025 | PMC12211446 | ci | All reported prevalence 95% CIs, including the headline 6.2% (6.18–6.22) in the abstract, are impossibly narrow. With n = 822, the correct binomial intervals span several percentage points (e.g. 6.2%: about 4.6–7.9%; 4.14%: about 2.8–5.5%). The paper therefore grossly misstates the precision of its |
| JAMA Netw Open 2026 | PMC13003376 | abstract_body_mismatch | The abstract says higher MELD was associated with 'lower referral odds', yet it reports OR 1.13 (95% CI 1.07-1.18). The Results, Table 2 and the descriptive data all show higher MELD was associated with increased odds of referral. This is a genuine direction reversal of a primary-outcome result in t |
| Nat Commun 2020 | PMC7679396 | ci | The repositive-rate CI (0.423–0.574%) does not contain the point estimate of 0.31% (107/34,424). No standard method could produce it; the correct exact CI is about 0.25–0.37%. The error appears in both the abstract and the Results, so it is a mathematically impossible reported key result in the abst |
| Proc Natl Acad Sci U S A 2022 | PMC8916005 | p_value | Genuine error. The reported chi-square P = 0.0379 is incompatible with the paper's own counts (19/45 vs 10/35), for which Pearson chi-square = 1.59, P = 0.21 (Yates P = 0.30; Fisher two-tailed P = 0.23). The reported value is exactly recoverable by running the test on the percentages as if each grou |
| Sci Adv 2026 | PMC12757054 | table_arithmetic | The chi-square statistics in Table 1 (and the analogous test in the Results text) cannot be derived from the reported animal counts: 4/16 vs 14/28 gives χ² = 2.63 (p ≈ 0.105), not χ² = 13.333 (P = 0.0003); 2/16 vs 9/28 gives χ² = 2.10 (p ≈ 0.15), not 10.351 (P = 0.0013); 4/16 vs 9/24 gives χ² = 0.68 |
| Nat Commun 2026 | PMC13351079 | claim_vs_data | The paper's Results report temperature as a significant NEGATIVE predictor of H7 diffusion at both origin (β = -0.484, 95% HPD -0.900 to -0.237) and destination (β = -0.227, -0.499 to -0.047), and the Discussion explicitly describes and mechanistically explains a negative temperature-transmission re |
| Hum Brain Mapp 2009 | PMC2903209 | p_value | The paired t statistics (-1.200, -1.495, -0.274, +0.175) match the Table I means and SDs. However, the corresponding p-values (0.001, 0.009) are impossible for these t values. The correct p-values are roughly 0.17–0.86, all non-significant. The p-values appear to have been copied from the baseline/f |
| Cell 2010 | PMC2887341 | abstract_body_mismatch | Genuine inconsistency: the abstract states that 'G3249A and G3244A' interfere with key sequence-recognition interactions and eliminate termination, whereas the Introduction, Results and Conclusions all consistently identify G3242A (not G3244A) as the second such mutation, and explicitly describe G32 |
| PLoS Pathog 2012 | PMC3261917 | abstract_body_mismatch | Genuine internal contradiction: the abstract claims arg1 knockdown 'led to increased generation of nitric oxide', whereas the Results heading, Results text, Fig. 6I legend, and Discussion all state that arg1 knockdown did NOT increase NO production, a point the authors rely on to argue a polyamine-m |
| Hepatology 2009 | PMC2759707 | claim_vs_data | Genuine and verified contradiction: the abstract states TIMP-1 and TIMP-2 increased after chronic CCl4 'only in the gp91phox-/- mice', whereas the Results (Fig. 7, p<0.05), the Discussion, and the paper's concluding mechanistic argument all state the increase occurred only in the WT mice and that gp |
| Int J Epidemiol 2018 | PMC6422062 | abstract_body_mismatch | The abstract says LBW prevalence rose from 9.0% to 10.1%. Table 1 data, which I recomputed, give 8.2% in 1982 and 8.3% in 2015. The Results, Discussion and Key Messages all say LBW stayed stable (P for trend 0.80). The abstract therefore reports a rising trend where the paper's data show none, which |
| BMJ Glob Health 2020 | PMC7061886 | claim_vs_data | The abstract and Results report OR 1.6 with a CI excluding 1 and p<0.001, yet state there was no statistically significant change in CFR. That is a direct self-contradiction in the headline result and conclusion. The OR is also inconsistent with the paper's own data: the CFR declined, so the OR shou |
| PLoS Comput Biol 2020 | PMC7048302 | claim_vs_data | Genuine error. Table 3 shows LabMol-149 with EC50 = 1.450 \u03bcM (3D7) and 0.509 \u03bcM (W2), i.e. 1450 nM and 509 nM, yet the Abstract and Discussion claim it shows "low nanomolar" activity (EC50 < 500 nM) and that it "completely satisfies" the antimalarial candidate criteria. No innocent explana |
| Psychol Med 2015 | PMC4682193 | claim_vs_data | The abstract calls the primary-outcome reduction in suicidal behaviour 'significantly greater'. The reported result is p=0.162 with a 95% CI (-1.71 to 0.09) that crosses zero, which is non-significant even at the trial's own lenient two-sided alpha of 0.15. The abstract also mislabels the raw treatm |
| Clin Cancer Res 2008 | PMC2907116 | ci | The error is genuine. Both reported hazard-ratio confidence intervals (2.0-3.1 for HR 2.5; 4.1-4.8 for HR 4.5) imply test statistics of z~8 and z~37, utterly incompatible with the accompanying p-values of 0.026 and 0.046; back-calculation from the p-values gives CIs of roughly 1.1-5.6 and 1.0-19.8. |
| Nat Commun 2018 | PMC5998138 | claim_vs_data | The abstract claims reduced PKN1 and PKN4 in CFBE41o- cells. The results show increased PKN2, confirmed by Western blot, and only ambiguous evidence for PKN1; PKN4 never appears. This is a headline abstract claim that contradicts the results in both direction and protein identity, so it is a major e |
| JAMA Netw Open 2025 | PMC12406064 | claim_vs_data | The reported t129 = 0.71 is incompatible with P < .001; it corresponds to P ≈ 0.48. The heavily overlapping CIs and the 1.1-point difference confirm the comparison is nonsignificant (P ≈ 0.3–0.5). The abstract and Results nonetheless call the HDRS-17 reduction 'significantly' greater, and this claim |
| Proc Natl Acad Sci U S A 2022 | PMC8916005 | p_value | The error is genuine and reproducible. For 19/45 (42%) vs 10/35 (28%), the 2x2 Pearson χ² is 1.59 (P = 0.21; Yates P = 0.31; Fisher P ≈ 0.23), not P = 0.0379. The published value is exactly what one obtains by applying χ² to the rounded percentages as if they were counts out of 100 (χ² = 4.31, P = 0 |
| Lancet Reg Health Eur 2025 | PMC12859599 | abstract_body_mismatch | The abstract presents the univariate HRs as the adjusted multivariate result and assigns them to the wrong treatments (immunotherapy 0.470 vs the body's targeted 0.470). The body and Discussion give adjusted HRs of 0.768 for immunotherapy and 0.556 for targeted therapy. The direction of the conclusi |
| PLoS Comput Biol 2025 | PMC11975116 | abstract_body_mismatch | The abstract's headline claim that WGS spanned '49 (98%)' of the 48 P-VNTRs is impossible. It also contradicts the Results (92%) and the Discussion (45 of 48, matching the three named poorly covered loci). This is a real abstract–body mismatch in a headline comparison of the paper. The qualitative c |
| Neurology 2026 | PMC13445500 | abstract_body_mismatch | The abstract reports that 73.5% (175/238) of trials used at least one biomarker. The body (Results, the phase sums 79+55, and the Discussion) consistently gives 56.3% (134/238). The abstract figure appears to be the clinical-outcome-as-primary statistic copied into the wrong place. This is a genuine |
| Psychol Med 2026 | PMC12902173 | sample_size | The analytic sample sizes conflict irreconcilably. Table 2 and the text give 361,258 participants for Cohort 2, while the flowchart gives 32,220, and the reported event percentages match only the flowchart. Cohort 1 shows the same pattern (468,722 vs 36,804), and its 'prior IBS excluded' cohort cont |
| Nat Commun 2023 | PMC9990026 | p_value | The flag is correct. The headline acquisition result (3/7 vaccinated vs 0/6 control uninfected after 9 challenges) gives a one-sided Fisher exact p of 35/286 = 0.12 (0.091 if the figure's n=6 vaccinated is used), i.e., non-significant. The reported 'P < 0.0001' is not merely wrong but unattainable b |
| PLoS Pathog 2023 | PMC10449197 | abstract_body_mismatch | Genuine error. All internal evidence (Beta = +1.00 for B.1.177, Table 1's 2.15 factor, the Results section heading, the sensitivity analysis, and the Discussion's '115% higher among Alpha') shows Alpha had ~2.15-fold (115%) HIGHER viral burden than B.1.177, equivalent to B.1.177 being ~54% lower. Th |
| Commun Biol 2025 | PMC12855784 | other | The error is genuine and not explainable by rounding, alternative settings, or subgroup/denominator differences. Table 1 (P41212, 28.81–1.45 Å, 1 molecule per ASU, 733 macromolecular non-H atoms, Wilson B 20.9, average B 23.55) is internally self-consistent with a single ~115-residue chain in a tetr |
| Eur Respir J 2023 | PMC10600351 | abstract_body_mismatch | The abstract reports p<0.001 for the CF versus rheumatoid arthritis matched comparison. Table 4 gives p=0.055, and the reported HR and CI (1.21, 1.00–1.48) imply p≈0.056. The abstract p-value is wrong, likely copied from the unadjusted column. The text's claim that CF risk was 'greater than' RA is n |
| Nat Commun 2019 | PMC6520353 | claim_vs_data | Genuine error: the abstract swaps the two human heritability estimates, stating that host variation explains 'almost half' of susceptibility and 'one-third' of severity, whereas the Results, Table 4 and Discussion consistently report 29% \u00b1 7% for susceptibility and 49% \u00b1 14% for severity ( |
| Proc Natl Acad Sci U S A 2018 | PMC6176606 | claim_vs_data | Genuine internal contradiction. The abstract states that the NS3-G459L variant 'dominantly inhibited negative-strand RNA synthesis' of WT virus, whereas the Results explicitly report that negative-strand replicon RNA was significantly above the GDD baseline (negative-strand synthesis permitted) whil |
| Kidney Int 2011 | PMC3394593 | claim_vs_data | Genuine error confirmed. The abstract sentence 'Only TLR4 (-/-) leukocytes produced IL6 in vivo and in response to HMGB1 in vitro' reverses the genotype: the Results (Figures 9 and 10), both figure legends, and three separate Discussion points all state that only TLR4 (+/+) leukocytes/macrophages pr |
| JAMA Netw Open 2023 | PMC9856961 | ci | This is a genuine error. The headline HbA1c effect of 0.02% lies outside its own reported 95% CI (0.03-0.14), in both the abstract and Table 3. The CI and the raw changes point to an effect of roughly 0.09 percentage points, and the reported SE and p value are also inconsistent with the estimate. Be |
| J Neurosci 2010 | PMC2874744 | abstract_body_mismatch | The flagged inconsistency is genuine. The abstract states that gain-of-miR-15a function reduces and loss-of-miR-15a function increases OGD-induced CEC death ('respectively'), which is the exact opposite of the Results (Fig 5A–D: mimic aggravates death and caspase-3 activation; inhibitor alleviates t |
| Kidney Int 2021 | PMC7927648 | ci | The point estimate 8.4 lies outside its own reported 95% CI (4.4–6.7), which is mathematically impossible. The same values appear in the abstract, Results and Table 4. This is a genuine error, most likely a mistyped upper bound. Significance is probably not overturned, but the CI for a headline resu |
| Gastroenterology 2020 | PMC7857017 | claim_vs_data | Genuine error. The abstract states 'Lower DDIT4 expression correlated with longer survival of gastric cancer patients,' which is the reverse of what is reported everywhere else in the paper (Introduction: 'Patients with low expression have poorer survival'; Results: 'statistically significant correl |
| JAMA Psychiatry 2015 | PMC5103694 | ci | The reported OR of 0.96 lies outside its own 95% CI of 0.01–0.75 and is inconsistent with P = .03. The CI and P instead imply an OR of about 0.09. The figure is impossible as reported and appears in the abstract and in the headline predictive result. It looks like a transcription error, since the co |
| JAMA Netw Open 2024 | PMC11579790 | claim_vs_data | The Key Points and abstract Conclusions state that JAKis had a statistically significantly increased cancer risk versus TNFis. The reported HR of 1.36 (95% CI, 0.94-1.96) includes 1, and the paper's own Results, Discussion and main Conclusions say the JAKi association was not significant. This misst |
| Lancet Reg Health Eur 2025 | PMC11910110 | abstract_body_mismatch | The abstract attributes the low birth weight estimates for ADHD, ASD and non-verbal IQ to preterm birth. For ASD this reverses the result: the body states preterm birth was not associated with ASD (0.60, CI crossing 0), yet the abstract reports a significant preterm–ASD association. The externalizin |
Preregistration track: trials vs their ClinicalTrials.gov registration
7,184 trial reports screened by GPT-6 Luna (high) and Claude Haiku 5.5 (high); 947 where both flag an undisclosed major discrepancy went to Opus 5.5 (high), which confirmed 280 major and 554 minor (48 that Opus 5.5 refused were verified by Opus 5 instead).
Confirmed major discrepancies
| Journal | Paper / registration | Summary (Opus verifier) |
|---|---|---|
| BMC Med 2026 | PMC13081374 NCT05929183 | The paper states that the trial was prospectively registered (NCT05929183). The registry shows the record was first submitted on 26 June 2023 and posted on 3 July 2023. By the paper's own account, recruitment began in November 2022 and the study ran until 30 June 2023, so the registration was made essentially at the end of the trial and the prospective-registration claim is inaccurate. |
| JAMA Netw Open 2024 | PMC10980970 NCT05415085 | The paper states the trial was prospectively registered. ClinicalTrials.gov shows the registration was first submitted on 2022-04-27, about a year after enrollment began in April 2021 and near the end of recruitment. The registration was therefore retrospective, and the paper's prospective-registration claim is incorrect and undisclosed. |
| Diabetologia 2024 | PMC11604790 NCT06115460 | The ClinicalTrials.gov record lists change in time in range (TIR) at 3 months as the sole primary outcome and microalbuminuria as secondary. The publication instead names change in UACR as the primary outcome, powers the study on it, and lists TIR as secondary, without acknowledging the change. Both outcomes are reported and both favour sitagliptin, but this is an undisclosed swap of the primary outcome; the registration was also submitted retrospectively, after primary completion. |
| BMC Med 2025 | PMC12217910 NCT05827822 | The ClinicalTrials.gov record names average risky drinking days per month as the sole primary outcome, with PACS craving as a secondary outcome assessed at two weeks. The published report instead presents PACS craving from baseline to 8 weeks post-intervention as its primary outcome, and reports risky drinking days (non-significant, P = 0.13) as secondary. The paper does not acknowledge this change. |
| JAMA Netw Open 2026 | PMC12980234 NCT06276569 | The ClinicalTrials.gov record lists nine primary outcomes, including SpO2/FiO2, inflammatory and myocardial markers, APACHE II and Murray scores at days 1-7, length of stay, and 30- and 90-day mortality; it does not list ARDS incidence. The published report designates ARDS incidence as its sole primary outcome and recasts the registered primaries as secondary, without acknowledging any change. Several registered measures are reported differently or not at all: PaO2/FiO2 instead of SpO2/FiO2, 48-hour worst scores instead of days 1-7, and no 30-day mortality. |
| Cell Rep Med 2026 | PMC13522775 NCT05459441 | The ClinicalTrials.gov record lists clinical response rate (CR+PR at 24 months) as the sole primary outcome and adverse events as a secondary outcome. The publication instead names dose-limiting toxicity incidence as the primary endpoint and presents response as secondary or exploratory. The paper does not acknowledge or explain this change. Response rates are still reported, but they are framed as exploratory. |
| BMC Med 2018 | PMC6038204 NCT00700895 | The registration lists a single primary outcome \u2014 the number of warfarin dose titrations required to achieve target INR assessed over 3 months (compiled every 4 weeks) \u2014 whereas the published report designates the number of dose titrations in the first 2 weeks (day 14) as the primary efficacy endpoint, with the sample size, non-inferiority margin and abstract headline result all based on that 14-day window. The 3-month titration count is reported, but as a secondary outcome. The paper discloses a 2016 protocol amendment adding time-in-therapeutic-range as a secondary endpoint but does not mention or justify the change in the primary outcome's time frame, and the registry entry was not updated accordingly. |
| Circulation 2021 | PMC8849598 NCT03476785 | The paper states the trial was registered prospectively, but the ClinicalTrials.gov record (NCT03476785) was first submitted in March 2018, nearly three years after the listed study start of June 2015. The registered enrollment (56) matches the trial, and the paper does not acknowledge the late registration. The registered primary outcome (LV stiffness at 1 year) is consistent with what was reported. |
| Am J Public Health 2009 | PMC2692954 NCT00314028 | The ClinicalTrials.gov registration lists frequency of unprotected penile-vaginal sex at 3 months as the sole primary outcome. The publication instead names subsequent STD diagnosis over a 6-month record review as the primary outcome and reports unprotected sex as a secondary behavioural outcome, which was not significant in adjusted and imputed analyses. The paper does not acknowledge or explain this difference from the registration. |
| JAMA Netw Open 2018 | PMC6324427 NCT01767883 | The ClinicalTrials.gov record lists the primary outcome as the percentage of patients meeting guideline-concordant care goals (BP, LDL, HbA1c, ACE/ARB use, NSAID avoidance, nephrologist referral, smoking cessation). The published report instead names annualized eGFR loss as its primary outcome, a measure absent from the registration. The paper does not acknowledge this difference, and it reports only some components of the registered measure as separate secondary outcomes, with no composite result. |
| PLoS Med 2021 | PMC8423266 NCT03518151 | The ClinicalTrials.gov record (NCT03518151) lists women's dietary quality score at 3 months as the sole primary outcome, with store sales of fruit/vegetables and confectionery as secondary outcomes. The published paper designates store sales as the primary outcome and dietary measures as secondary, without acknowledging or explaining the change. The registered primary outcome is still reported, but it is presented as secondary. |
| JAMA Netw Open 2026 | PMC13539392 NCT06033365 | The paper states the trial was registered before enrollment began, but the registry shows it was first submitted on April 25, 2023, about two weeks after the paper's reported enrollment start of April 10, 2023, and was posted publicly in September 2023. The registered primary outcome was also narrower: outpatient addiction treatment attendance at 30 days. The paper's primary outcome is a composite measured at both 30 and 90 days, and this difference from the registration is not acknowledged, although both results were null. |
| JAMA Netw Open 2024 | PMC11031685 NCT04191837 | The trial registration lists NRS pain at week 4 as the sole primary outcome, with week 8 and week 12 as secondary time points. The published abstract instead designates NRS pain at 12 weeks as primary, without acknowledging the change. The registered week-4 result is reported and also statistically significant, so the main conclusion is not reversed, but the primary time point was changed without disclosure. |
| JAMA Netw Open 2024 | PMC11074813 NCT04260724 | The ClinicalTrials.gov record lists the 4-week change in ADAS-Cog (among nine 4-week cognitive measures) as primary and the 8-week ADAS-Cog change as secondary. The publication names the 8-week ADAS-Cog change as the primary outcome and the 4-week change as secondary, without noting or explaining the change. The 4-week result was also statistically significant, but the switch in primary time point is undisclosed. |
| BMC Med 2024 | PMC11229192 NCT05503641 | Three of the six primary outcomes registered for NCT05503641 (dynamometer handgrip strength, the Lawton & Brody IADL scale and the Barthel Index) do not appear in the published trial report. The paper gives no explanation or disclosure, and it reports several unregistered outcomes in their place. There are also inconsistencies in reported group sizes (98 randomised, 95 analysed, Table 2 headers showing n=26/24), though these are of lesser weight. |
| BMC Med 2024 | PMC11406833 NCT04972032 | The registration specifies the primary outcome as the 60-day adhesion recurrence rate, with recurrence defined as any AFS score above 0 at second-look hysteroscopy. The publication instead reports as primary the rate of 'adhesion reduction', defined as a decrease of at least 4 AFS points, and does not report the registered AFS>0 recurrence rate. This change of primary outcome definition is not acknowledged or explained in the paper. |
| Diabetes Care 2024 | PMC11664189 NCT04545411 | The ClinicalTrials.gov registration lists five primary outcomes, including two measures of vascular endothelial function and measured HbA1c change. The published report presents neither endothelial outcome and gives only a CGM-derived estimated HbA1c, without explaining the omissions. The paper also used 7 days of CGM rather than the registered 2-week window; this is a smaller, secondary inconsistency. |
| JAMA Netw Open 2025 | PMC11866027 NCT04292379 | The paper states the protocol was registered before participant enrollment. The ClinicalTrials.gov record was first submitted on 27 February 2020, after the 29 January 2020 start of the preimplementation enrollment period described in the paper itself. The registered primary outcome (30-day child well-being) matches the published one, so the issue is limited to the accuracy of the prospective-registration claim. |
| Clin Cancer Res 2025 | PMC12353064 NCT02494869 | The ClinicalTrials.gov record lists only one primary outcome: gene expression patterns and H&E morphological features of normal breast epithelium over 2 years. The published report instead names change in Ki67 proliferation at week 25 as its primary endpoint and does not report the registered outcome. The paper does not acknowledge or explain the difference, so this is an undisclosed primary outcome change. |
| JAMA Netw Open 2025 | PMC13102256 NCT03965195 | The ClinicalTrials.gov registration lists all-cause hospitalization as the sole primary outcome. The published report designates respiratory-related (ICD-10 J-code) hospitalization as primary and calls all-cause hospitalization a secondary outcome, without acknowledging the change. All-cause hospitalization results are still reported and were null, like the new primary outcome. |
Deterministic registry checks (dates and identifiers compared by code, then verified by Opus 5.5):
9 cases
| Journal | Paper / registration | Check | Verdict | Summary |
|---|---|---|---|---|
| JAMA Netw Open 2024 | PMC10980970 NCT05415085 | false prospective claim | major | The paper states that the trial was prospectively registered as NCT05415085. The registry shows it was first submitted on 2022-04-27, roughly a year after enrollment began in April–May 2021 and close to the primary completion date. The registration is therefore retrospective, and the paper's claim of prospective registration is inaccurate. |
| Eur Respir J 2025 | PMC12332466 NCT02526004 | false prospective claim | minor | The paper calls CFMATTERS 'prospectively registered' as NCT02526004, but the registry shows first submission in August 2015, after recruitment began in January 2015. The paper itself acknowledges the delay in registration, and the registered primary outcome matches the reported one. The wording is therefore inaccurate but transparently qualified, making this a minor issue. |
| BMC Med 2026 | PMC13081374 NCT05929183 | false prospective claim | major | The paper says the trial was prospectively registered as NCT05929183. The registry shows it was first submitted on 26 June 2023, whereas the paper reports recruitment from November 2022 to 30 June 2023. The registration was therefore made retrospectively, near the end of the study, and the paper's claim of prospective registration is inaccurate. |
| BMC Med 2012 | PMC3292842 NCT00981877 | registration mismatch | minor | The ClinicalTrials.gov number cited in this paper (NCT00981877) belongs to an unrelated trial of probiotics for rotavirus diarrhea in Bolivian children, not to this iron-supplementation trial in Swiss blood donors. This looks like a citation error rather than evidence of outcome switching. However, it means the published report cannot be checked against its actual registration through the cited ID. |
| Eur Respir J 2012 | PMC3432241 NCT00759304 | registration mismatch | minor | One of the two ClinicalTrials.gov identifiers cited in this nicotine mouth spray trial (NCT00759304) points to an unrelated observational cohort study of ageing and cardiovascular events in 65-year-olds. This appears to be a mis-cited registration number. The second cited identifier may be the correct registration, and the error does not affect the reported results. |
| Eur Respir J 2012 | PMC3432241 NCT00766584 | registration mismatch | minor | One of the two ClinicalTrials.gov identifiers cited by this nicotine mouth spray trial, NCT00766584, belongs to an unrelated French observational cohort on aging, autonomic activity and sleep apnea. This appears to be a citation error, since the paper also lists a second identifier (NCT00759304) that may be the correct registration. The error affects traceability but not the reported results. |
| Lancet Diabetes Endocrinol 2016 | PMC4728199 NCT00797862 | registration mismatch | major | The ClinicalTrials.gov number cited for PATHWAY-3 (NCT00797862) belongs to a different trial: a 1254-patient aliskiren/amlodipine study (ACCELERATE) with a blood-pressure primary outcome. PATHWAY-3 is a 441-patient amiloride/hydrochlorothiazide trial with a 2 h glucose primary outcome. The cited registration therefore cannot be used to verify the trial's prespecified outcomes or registration timing; only the EudraCT number cited alongside it may be valid. |
| Circulation 2017 | PMC5459585 NCT01858363 | false prospective claim | major | The paper states that the trial was prospectively registered as NCT01858363. The registry, however, shows a first submission on 8 May 2013, after the November 2012 study start, and the paper itself reports enrollment beginning in December 2012. Registration therefore occurred roughly 5 months after enrollment began, and the prospective-registration claim does not hold. |
| Circulation 2021 | PMC8849598 NCT03476785 | false prospective claim | major | The paper states the trial was registered prospectively on ClinicalTrials.gov (NCT03476785). The registry record was first submitted on 2018-03-01, nearly three years after the listed study start of 2015-06-01 and about 16 months before primary completion. This indicates retrospective registration, and the paper does not acknowledge it. |
Compared against the current registry record (earlier registry versions are not available to us), so a registry edited after publication can hide or create discrepancies. Secondary analyses and follow-up reports are recognised and not penalised for omitting the primary outcome.
Top 5 examples so far
Reported significance of the in-vivo leak result cannot be reproduced from the paper's own counts
The Results state that CSF-leak frequency was significantly higher in all three Fbn2 mutant mouse strains than in wild-type mice ("all comparisons p<0·001"), with relative-risk CIs excluding 1. With the paper's own counts (15/19, 22/29, 8/11 vs 20/35) and the χ² test named in the Methods, p = 0.11, 0.12 and 0.36 (Fisher's exact 0.14–0.49), and the 95% CIs of the relative risks are 0.96–2.00, 0.93–1.89 and 0.80–2.02. The abstract's in-vivo claim of a predisposition to dural rupture rests partly on this comparison.
…frequencies were significantly higher for Fbn2A1052T/+ mice (15 [79%] of 19), RR 1·38, 95% CI 1·14–1·69), for Fbn2D1581V/+ mice (22 [76%] of 29], RR 1·33, 95% CI 1·09–1·63), and for Fbn2M2387T/+ mice (eight [73%] of 11], RR 1·27, 95% CI 1·04–1·58), with all comparisons p<0·001 (figure 4A).
Proportions of mice with leaks were compared between mutant and wild-type mice by use of χ2 tests.
Checks: Haiku full read → Sonnet verification → Opus defense + Opus judge → two fresh independent Opus checks → manual recomputation. No correction found in Crossref or PMC as of 2026-10-22.
Reported "risk ratios" match ratios of mean 1/2 codes, understating the effect roughly tenfold
The binary outcomes were coded 1 (no) and 2 (yes). The reported risk ratios equal the ratio of group means of those codes: at 3 months the reported RR is 0.978, (1+9/561)/(1+20/491) = 0.976, whereas the risk ratio from the same counts is 0.39. The pattern holds at every time point (reported 0.96–0.98 vs 0.39–0.67 from counts), so the Discussion's "about a 3% reduction in risk" understates relative reductions of roughly 33–61%.
…responses coded 1 for respondents not reporting ever experiencing the event and 2 for those reporting ever experiencing the event.
at 3 months (9 of 561 [2%] vs 20 of 491 [4%]; risk ratio [RR], 0.978; 95% CI, 0.959-0.997)
The magnitude of the intervention effects in this trial was modest, about a 3% reduction in risk.
Checks: Haiku full read → Sonnet verification → Opus defense + Opus judge → two fresh independent Opus checks → manual recomputation. No correction found in Crossref or PMC as of 2026-10-22.
A miscomputed registry proportion reverses a reported under-representation
The overall registry (CiNA) share of Asian or Pacific Islander patients is given as 5.9%, but the paper's own counts give 145,568/4,891,486 = 2.98%, consistent with its period-specific figures (2.7% and 3.4%). With either trial figure the paper reports (272/8,308 = 3.3% or 371/8,309 = 4.5%), Asian or Pacific Islander participants were over-, not under-represented overall, so the reported difference of −1.5 percentage points (P < .001) does not follow.
…an underrepresentation of Asian or Pacific Islander participants (trials, 272 of 8308 [4.5%]; CiNA, 145 568 of 4 891 486 [5.9%]; difference, -1.5 percentage points; P < .001)
Checks: Haiku full read → Sonnet verification → Opus defense + Opus judge → two fresh independent Opus checks → manual recomputation. No correction found in Crossref or PMC as of 2026-10-22.
An impossible confidence interval behind a "significant" stroke reduction
The Abstract and Results give the 9 µg/m³ stroke risk difference as −2.26 per mille (95% CI −8.97 to −20.64), an interval that excludes its own estimate. Table 2 gives −8.97 to +20.64, which includes zero, so the Key Points statement that the intervention would significantly reduce stroke hospitalization at this threshold is not supported by the paper's table. Two further CI sign discrepancies appear between abstract, text and table.
…the estimated 5-year risk difference of hospitalization for stroke was -2.26 per mille (95% CI, -8.97 to -20.64 per mille)
Table 2: ≤9 | 10.06 (3.37 to 32.97) | -2.26 (-8.97 to 20.64)
Checks: Haiku full read → Sonnet verification → Opus defense + Opus judge → two fresh independent Opus checks → manual recomputation. No correction found in Crossref or PMC as of 2026-10-22.
Implausibly narrow confidence intervals underpin associations reported in the abstract
With 76 catastrophic-expenditure cases among 296 participants, the abstract reports adjusted ORs of 2.0 (95% CI 1.9–2.1) for the second poverty quintile and 9.7 (9.6–9.8) for subcounty hospitals. The table counts imply standard errors 15–75 times larger (crude 95% CIs 0.52–2.81 and 4.1–19.5), and the unadjusted second-quintile OR is 1.4 (0.8–2.4), p = 0.30. The abstract also disagrees with Table 2 (9.6–9.8 vs 9.7–9.9).
…belonging to the second poverty quintile (2·0, 1·9–2·1, p<0·0001), and seeking care from subcounty hospitals (9·7, 9·6–9·8, p<0·0001)…
Table 2: Second quintile | 1·4 (0·8–2·4) | 0·30 | 2·0 (1·9–2·1) | <0·0001
Checks: Haiku full read → Sonnet verification → Opus defense + Opus judge → two fresh independent Opus checks → manual recomputation. No correction found in Crossref or PMC as of 2026-10-22.
Unverified candidates (13,688 flags in 11,030 papers)
Two-model agreement only (cheap cascade); not yet independently checked. Categories: claim_vs_data 5544 · abstract_body_mismatch 4977 · p_value 717 · ci 639 · table_arithmetic 594 · sample_size 399 · impossible_value 322 · pct_count 244 · other 223 · outcome_switch 28 · label 1.
Top 25 by confidence
| Journal | PMCID | Category | Problem (model summary) |
|---|---|---|---|
| PLoS Med 2022 | PMC9718399 | pct_count | The unadjusted rates are labelled as percentages, but the stated counts give rates of about 26.4 and 20.1 per 1,000, not per cent. Table 4 reports the same raw rates as 'Rate per 1,000 patients at risk'. The abstract's u |
| Lancet 2012 | PMC10782811 | abstract_body_mismatch | The abstract reports five states (11%) with weights below 0.01, but Table 2 contains nine state estimates below 0.01. The reported count and percentage are also inconsistent with the stated total of 220 states. |
| Clin Cancer Res 2008 | PMC2830553 | abstract_body_mismatch | Abstract states 3% of patient samples had single-locus MSI, but results show 8 of 52 samples (about 15%) were L-MSI; the 3% refers to markers. |
| Gastroenterology 2012 | PMC3572711 | ci | The 95% CI for rs3802842 does not contain its point estimate, and its upper bound is almost equal to its lower bound. This SNP is one of the eight headline associations listed in the abstract and described as very strong |
| Psychol Med 2013 | PMC3936825 | abstract_body_mismatch | The abstract reports age standard deviations that conflict with Table 1. The bipolar subgroups’ means and SDs imply an overall patient SD of about 6.0 years, not 11.8; the control SD is reported as 6.1, not 12.0. |
| Hepatology 2008 | PMC2768766 | p_value | The overall-survival Wald statistic and p-value are inconsistent, so the reported significance of XIAP as an independent predictor of overall survival is not supported by the stated test. |
| Sci Adv 2024 | PMC10776005 | sample_size | Abstract states 149 TB subjects, while Results and Methods state 103 TB subjects (149 is the total of TB plus non-TB). |
| Lancet Public Health 2025 | PMC12700172 | claim_vs_data | The abstract says the unadjusted no-debt difference was not significant, but the Results and Table 3 report unadjusted HR 1.48 with CI 1.07–2.05 excluding 1, i.e. significant. Also the crude IRR in Table 2 is 1.52 (1.21– |
| JAMA Netw Open 2023 | PMC9982699 | pct_count | Incident IS count in abstract (15.5 million) is inconsistent with 3.4 million total incident strokes and with the 86.8% share; also 3458 of 4449 IS cases is 77.7%, not 86.8%. |
| JAMA Netw Open 2023 | PMC9982699 | abstract_body_mismatch | The subtype estimates are described as incident strokes, although their total and percentages correspond to the 17.8 million prevalent cases, not the 3.4 million incident cases. |
| PLoS Med 2020 | PMC7605626 | abstract_body_mismatch | The abstract's ICER values do not match the Results text and Table 2 for either time horizon. |
| Lancet 2012 | PMC10782811 | pct_count | The percentage in the abstract does not match its count. Five states out of the 220 analysed is about 2%, not 11%. An 11% figure would imply a denominator of about 45 states, which is inconsistent with the 220 states use |
| Nat Genet 2021 | PMC7611517 | abstract_body_mismatch | The abstract says no significant effect was observed following maternal BPA exposure, but the Results report a statistically significant increase at IAP-Rnf157 in male offspring exposed to the upper BPA dose. |
| Nat Commun 2026 | PMC13526883 | abstract_body_mismatch | The abstract calls more than 100 channels “usable,” whereas the body distinguishes more than 100 channels observed above the noise floor from 42 channels whose BER meets the stated threshold; it says the total number of |
| JAMA 2025 | PMC12687205 | pct_count | The stated percentages do not equal the numerator/denominator given for either endpoint year. The same 2.92% and 5.18% are repeated in the Discussion. |
| JAMA 2025 | PMC12687205 | pct_count | The reported endpoint percentages do not match their stated numerators and denominators. |
| J Neurosci 2010 | PMC2923025 | p_value | F(1,12)=4.66 corresponds to p around 0.05, not p<0.0001; the reported p-value is impossible for this F statistic. |
| Diabetologia 2017 | PMC6002965 | abstract_body_mismatch | The CI widths for the duration of hypoglycaemia differ drastically between abstract and results; the abstract CIs are implausibly narrow (about ±12 min) compared with the results CIs (about ±300 and ±217 min) for the sam |
| Clin Cancer Res 2008 | PMC2830553 | pct_count | Single-locus instability counts in the newly diagnosed group are described as 4 of 25 samples, but the 3% figure is then attributed to both groups; the abstract reports 3% of patient samples with single-locus MSI, while |
| Clin Cancer Res 2008 | PMC2830553 | abstract_body_mismatch | The abstract reports 3% of patient samples with single-locus MSI, but the results report eight affected samples across the two groups. |
| PLoS Med 2017 | PMC5360219 | claim_vs_data | The discussion reverses the direction of the PHS association described in the results: higher PHS is associated with earlier, not later, expected onset. |
| PLoS Med 2019 | PMC6445413 | abstract_body_mismatch | Abstract says INH and KAN range upper bound 0.73, but Table 3 lists INH fungal ball RPL 3.04 and KAN fungal ball 0.918, and the body text says INH penetration was higher in caseous nodules and fungal ball concentrations |
| Proc Natl Acad Sci U S A 2023 | PMC10691225 | table_arithmetic | Persistence index is defined as a ratio of displacement to path length, which must be ≤1 by definition; reported values of 1.67 and 2.67 are impossible. |
| Proc Natl Acad Sci U S A 2023 | PMC10691225 | impossible_value | The reported persistence-index values are impossible under the paper’s stated definition. |
| Nucleic Acids Res 2008 | PMC2425497 | abstract_body_mismatch | The abstract’s before-calibration accuracy does not agree with the five resolved targets’ Table 4 accuracies weighted by their reported n. The after-calibration accuracy does agree after rounding. |
Each top example passed: cheap full read → Sonnet verification with recomputation → Opus defense and Opus judge → two fresh independent Opus checks (reproduction; materiality and existing-correction search) → manual re-derivation. Wording is factual; no claim about intent is made.