Citation benchmark
Out-of-date law is the real risk now.
A current frontier model rarely invents CFR section numbers, but it states rules removed in February 2026 as current law. Compliome labels every removed section as former, points to today's rule, and links about 21 official sources per answer.
Results
Both systems answered the same 78 questions. The baseline is the same model that powers Compliome, Claude Sonnet 5, with no tools and an instruction to cite only real sources.
| Metric | Compliome | Bare model |
|---|---|---|
| CFR citations in answers | 460 | 176 |
| Exist in today's CFR (eCFR, U.S. government source) | 95.0% | 89.8% |
| Removed by the 2026 QMSR (Part 820 rewrite, effective 2 Feb 2026) | 23 | 17 |
| … of which labeled as former / historical | 23 / 23 | 0 / 17 |
| Fabricated (section never existed) | 0 | 1 (21 CFR 860.219) |
| Sources resolved to an official URL | 1,626 (20.8 per answer, 19 source types) | 0 |
| Non-US sources with a live official URL | 803 / 804 | None produced |
Run 11 October 2026. Bare-model answers were capped at 700 tokens, so it made fewer citations overall.
One example: design controls
On 2 February 2026 the Quality Management System Regulation (QMSR) replaced most of 21 CFR Part 820 and incorporated ISO 13485:2016 by reference. Former § 820.30, "Design controls", no longer exists.
"Design controls are established under 21 CFR § 820.30."
Stated as current law. That section was removed on 2 February 2026.
Design and development requirements now sit in § 820.10, through ISO 13485:2016 clause 7.3.
When it discusses older warning letters that cite § 820.30, it labels the section as former.
What this means
- A current frontier model rarely invents CFR section numbers. The bare model made one fabricated CFR citation in 176. Invented citations are not the main problem any more.
- Its real failure is out-of-date law presented as current. All 17 of its superseded citations are Part 820 sections the QMSR removed, stated as live requirements. Compliome cites removed sections only when discussing pre-2026 warning letters and inspection findings, and labels every one as former (23 of 23).
- Zero fabricated CFR citations from Compliome. Every one of its 460 CFR citations either exists today or is a removed Part 820 section clearly labeled as former.
- The biggest gap is verifiability and breadth. Each Compliome answer carries about 21 resolved sources (CFR, guidance, warning letters, 483s, recalls, labels, trials, EU/UK/Canadian/Australian law, ICH, MHRA) with official links. The bare model's citations are text you have to look up yourself, and it produced no EU, UK, Canadian or Australian links at all.
Methodology
- Questions. 78 fixed regulatory questions spanning US regulations (21 CFR, the Common Rule in 45 CFR 46, and 42 CFR 11 for ClinicalTrials.gov), FDA guidance, enforcement, drugs, devices, clinical trials, EU, UK, Canadian and Australian legislation, ICH guidelines and MHRA guidance.
- Systems. Compliome was run through its production API, one answer per question. The baseline was Claude Sonnet 5 with no tools, told to cite only real sources.
- Oracle. A CFR citation counts as real only if the section exists in the current CFR, checked against the eCFR structure API, the U.S. government's own source. The same check is applied to both systems.
- Superseded vs fabricated. A citation missing from today's CFR is superseded if the section existed in the CFR as of 1 December 2025 (before the QMSR took effect), and fabricated if it did not exist then either.
- Official links. For Compliome we also checked that every non-US source resolves to a live page on the official site (legislation.gov.uk, EUR-Lex, laws-lois.justice.gc.ca, legislation.gov.au, GOV.UK).
Caveats
- It checks that a cited section exists, not that it supports the claim. A real section can still be cited for the wrong point. This is a citation-integrity benchmark, not a measure of overall answer quality.
- One model at one point in time. Results for other models, or for this model later, may differ.
- Shorter baseline answers. Bare-model answers were capped at 700 tokens, so it made fewer citations (176 vs 460).
- We ran it ourselves. The benchmark script is in our repository and the run is reproducible, but it has not been independently audited.
Check the citations yourself.
Free during the beta · No card required