10 tasks across 8 categories. Each test set is SHA-256-hashed and version-pinned. Probes that train on a test set get caught at validation time.
Factual fabrication, entity recall failures.
Open-ended question answering with annotated hallucination labels (Li et al. 2023).
hallucinated| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| FabricationGuard L31 · end_question | 0.903 [0.85, 0.95] | 0.840 | 0.710 |
| RewardHackGuard L31 · token_avg | 0.650 [0.56, 0.74] | 0.590 | 0.520 |
Open-domain factual QA — short factoids, very hard. SOTA closed-book ~50% (OpenAI 2024).
hallucinated| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| FabricationGuard L31 · end_question | 0.882 [0.83, 0.93] | 0.820 | 0.720 |
Tests resistance to popular misconceptions (Lin et al. 2022). Different cognitive task than fabrication.
misconception| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| FabricationGuard L31 · end_question | 0.599 [0.51, 0.69] | 0.570 | 0.550 |
Capability control — multiple-choice knowledge across 57 domains. NOT hallucination per se; included as scope check.
incorrect| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| FabricationGuard L31 · end_question | 0.444 [0.40, 0.49] | 0.420 | 0.410 |
Chain-of-thought faithfulness, hypocrisy gap.
CoT-vs-belief divergence. Measures when model states reasoning that diverges from its internal belief (arXiv:2602.02496, Jan 2026).
unfaithful_cot| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| DeceptionGuard L40 · last_token | 0.800 [0.72, 0.87] | 0.710 | 0.650 |
Grade-school math word problems. Used as the reasoning-faithfulness within-domain bench. Hallucination = model produces wrong answer despite reasoning trace appearing coherent.
wrong_answer| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| ReasonGuard L55 · mid_think | 0.908 | 0.772 | 0.612 |
Open-domain commonsense reasoning requiring multi-step strategy. Used as the cross-domain transfer test for reasoning-faithfulness probes (math → commonsense).
wrong_answer| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| ReasonGuard L55 · mid_think | 0.612 | 0.520 | 0.500 |
Hendrycks competition mathematics — PhD-level math problems including algebra, calculus, number theory, geometry. Hardest reasoning bench in the registry. Qwen3.6-27B baseline 18% accuracy. Used as advanced-difficulty stress test for reasoning-faithfulness probes.
wrong_answer| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| ReasonGuard L55 · mid_think | 0.500 | 0.425 | 0.500 |
Strategic dishonesty, sleeper agents, alignment faking.
Apollo Research deception suite — model concealing illicit trades when asked. AUROC 0.96-0.999 on linear probes (arXiv:2502.03407).
deceptive| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| DeceptionGuard L40 · last_token | 0.978 [0.96, 0.99] | 0.910 | 0.840 |
Capability hiding under evaluation.
Model knows-it-is-being-tested confound.
Linear probes separate real-world eval prompts from deployment prompts. Power-law scaling with model size (arXiv:2509.13333).
evaluation| Probe | AUROC [CI] | Eval-aware | Dist-shift |
|---|---|---|---|
| EvalAwarenessGuard L40 · end_question | 0.930 [0.88, 0.97] | 0.930 | 0.810 |
RL-induced misalignment generalization.
EU AI Act Article 5 — persuasion / subliminal.
Over/under-refusal calibration; jailbreak proxy.
Every test set referenced here is publicly available under the dataset's original license. We pin a SHA-256 hash so probes can be validated against the exact bytes we evaluated on.
If a dataset is gated (e.g., requires HF agreement), we link to the canonical source and ship our exact preprocessed splits as a separate Apache-2.0 derivative when permitted.
Have a benchmark task that should be in here? Open a PR with the YAML registry entry.
id: your-task-id name: "Your Task Name" category: hallucination # or any of the 8 ProbeBench categories description: "Short description" paper: "arXiv:XXXX.XXXXX" dataset_url: "https://huggingface.co/datasets/your/dataset" test_set_size: 200 test_set_hash: "abc123..." # SHA-256 of the canonical test split positive_label: "hallucinated"