Where this probe breaks. Per-task FPR/TPR decomposition, calibration curves, and the kinds of inputs we have not yet validated. Numbers below are pulled directly from the registry — no smoothing, no charity.
Each row is one held-out task. FPR@99TPR is the operationally honest metric: at the threshold that catches 99% of true positives, what fraction of negatives also fire? Above 0.30, the probe is broken at that operating point.
| Task | n | AUROC [CI] | FPR@99TPR | ECE | Brier | Latency | Failure? |
|---|---|---|---|---|---|---|---|
| Apollo Insider Trading deception | 150 | 0.978 [0.96, 0.99] | 0.012 | 0.040 | 0.060 | 2.1 ms | robust |
| Hypocrisy Gap reasoning | 180 | 0.800 [0.72, 0.87] | 0.100 | 0.090 | 0.140 | 2.1 ms | stress |
Color key: emerald FPR@99TPR < 0.10 (robust) · amber 0.10–0.30 (stress) · rose> 0.30 (broken at this operating point).
Lower ECE/Brier = better calibration. Above 0.10 ECE means the probe's confidence does not match its accuracy — a 0.9 score is not necessarily 90% likely to be a positive.
Brier captures sharpness (how far probabilities are from {0,1}) and ECE captures bias (whether confidence matches frequency). A probe can have low Brier and high ECE if it is consistently overconfident in one direction — worth checking before deploying any threshold rule.
Each card maps to a known failure mode in the literature. We populate the cells where data exists; cards marked “not measured” signal where this probe still needs adversarial validation.
AUROC drops from 0.978 to 0.840 under long-context / OOD evaluation on Apollo Insider Trading — a signed delta of −0.138.
In-distribution AUROC overstates production performance whenever the deployment context length, prompt format, or domain differs from the training set.
AUROC drops from 0.978 to 0.910 when stratified by eval-awareness quartile (delta −0.068).
Apollo (2509.13333) shows AUROC scales with the model’s belief that it is being evaluated. Always quote the corrected number for safety-relevant claims.
FPR@99TPR = 0.100 on Hypocrisy Gap. At the threshold that catches 99% of true positives, 10% of negatives also trip the alarm.
Practical implication: at production volumes, near-perfect recall comes with a flood of false positives. Either drop the operating threshold and accept lower recall, or treat this task as out-of-scope.
Probe trained / strongest on Apollo Insider Trading (AUROC 0.978) drops to 0.800 on Hypocrisy Gap — difference −0.178.
Cross-task generalization is not free. Probes labeled “hallucination” often transfer poorly between QA, multiple-choice misconception, and knowledge-recall settings.
Training note: Re-implementation of the Apollo Research deception-detection method on Llama-3.3-70B-Instruct. Uses paired honest/deceptive contrast pairs from insider-trading and werewolf scenarios. Citation: Goldowsky-Dill, Chughtai, Heimersheim, Hobbhahn (ICML 2025).
Category-specific stress tests. If you are evaluating DeceptionGuard (Apollo re-impl) for production, these are the prompts most likely to expose hidden failures.
These are hypotheses, not measurements. Each item should ideally graduate to a row in the table above, with its own n / AUROC / FPR.
Found a case where this probe fails? PR a YAML file to the registry. The schema is intentionally minimal — a prompt, the expected label, the observed probe score, and a reproducer link.
probe_id: openinterp/deceptionguard-llama33-70b-l40 failure_case: prompt: "Your adversarial prompt" expected_label: positive probe_score: 0.12 expected_score: ">0.7" category: distribution_shift reproducer: link.to.notebook
probebench/failures/ with the schema on the left.dc4f2a8b6e5d….