Where this probe breaks. Per-task FPR/TPR decomposition, calibration curves, and the kinds of inputs we have not yet validated. Numbers below are pulled directly from the registry — no smoothing, no charity.
Each row is one held-out task. FPR@99TPR is the operationally honest metric: at the threshold that catches 99% of true positives, what fraction of negatives also fire? Above 0.30, the probe is broken at that operating point.
| Task | n | AUROC [CI] | FPR@99TPR | ECE | Brier | Latency | Failure? |
|---|---|---|---|---|---|---|---|
| HaluEval-QA hallucination | 200 | 0.903 [0.85, 0.95] | 0.040 | 0.080 | 0.130 | 1.0 ms | robust |
| SimpleQA hallucination | 100 | 0.882 [0.83, 0.93] | 0.050 | 0.070 | 0.120 | 1.0 ms | robust |
| TruthfulQA-MC1 hallucination | 200 | 0.599 [0.51, 0.69] | 0.920 | 0.180 | 0.240 | 1.0 ms | broken |
| MMLU hallucination | 500 | 0.444 [0.40, 0.49] | 0.990 | 0.220 | 0.270 | 1.0 ms | broken |
Color key: emerald FPR@99TPR < 0.10 (robust) · amber 0.10–0.30 (stress) · rose> 0.30 (broken at this operating point).
Lower ECE/Brier = better calibration. Above 0.10 ECE means the probe's confidence does not match its accuracy — a 0.9 score is not necessarily 90% likely to be a positive.
Brier captures sharpness (how far probabilities are from {0,1}) and ECE captures bias (whether confidence matches frequency). A probe can have low Brier and high ECE if it is consistently overconfident in one direction — worth checking before deploying any threshold rule.
Each card maps to a known failure mode in the literature. We populate the cells where data exists; cards marked “not measured” signal where this probe still needs adversarial validation.
AUROC drops from 0.903 to 0.710 under long-context / OOD evaluation on HaluEval-QA — a signed delta of −0.193.
In-distribution AUROC overstates production performance whenever the deployment context length, prompt format, or domain differs from the training set.
AUROC drops from 0.903 to 0.840 when stratified by eval-awareness quartile (delta −0.063).
Apollo (2509.13333) shows AUROC scales with the model’s belief that it is being evaluated. Always quote the corrected number for safety-relevant claims.
FPR@99TPR = 0.990 on MMLU. At the threshold that catches 99% of true positives, 99% of negatives also trip the alarm.
Practical implication: at production volumes, near-perfect recall comes with a flood of false positives. Either drop the operating threshold and accept lower recall, or treat this task as out-of-scope.
Probe trained / strongest on HaluEval-QA (AUROC 0.903) drops to 0.444 on MMLU — difference −0.459.
Cross-task generalization is not free. Probes labeled “hallucination” often transfer poorly between QA, multiple-choice misconception, and knowledge-recall settings.
Training note: Multi-feature L2 logistic regression on residual stream at L31 of Qwen3.6-27B. Trained cross-bench on TruthfulQA + HaluEval + MMLU train splits, evaluated cross-task on held-out SimpleQA. Ships in openinterp PyPI v0.2.0.
Category-specific stress tests. If you are evaluating FabricationGuard v2 for production, these are the prompts most likely to expose hidden failures.
These are hypotheses, not measurements. Each item should ideally graduate to a row in the table above, with its own n / AUROC / FPR.
Found a case where this probe fails? PR a YAML file to the registry. The schema is intentionally minimal — a prompt, the expected label, the observed probe score, and a reproducer link.
probe_id: openinterp/fabricationguard-qwen36-27b-l31-v2 failure_case: prompt: "Your adversarial prompt" expected_label: positive probe_score: 0.12 expected_score: ">0.7" category: distribution_shift reproducer: link.to.notebook
probebench/failures/ with the schema on the left.fb8c2a4e1f9d….