DeceptionGuard (Apollo re-impl)
Re-impl of Apollo Research deception probe (AUROC 0.96–0.999 published)
Re-implementation of the Apollo Research deception-detection method on Llama-3.3-70B-Instruct. Uses paired honest/deceptive contrast pairs from insider-trading and werewolf scenarios. Citation: Goldowsky-Dill, Chughtai, Heimersheim, Hobbhahn (ICML 2025).
Quickstart
Three lines via the openinterp.probebench SDK. Returns the probe's P(positive_class) for a tensor of activations captured at the layer/position below.
from openinterp import probebench
probe = probebench.load("openinterp/deceptionguard-llama33-70b-l40")
score = probe.score(activations) # → P(positive_class)ProbeScore — 7 weighted axes
Composite metric in [0, 1]. No single axis can dominate; we think this discourages single-metric optimization. Numbers update as new evaluations land.
ProbeScore — 8-axis breakdown
ProbeBench v0.0.2 · 8 axes weighted to sum 1.0Visual breakdown of where this probe scores high vs low across all 8 axes. Polygon area is proportional to total score; shape reveals which axes carry it (and which ones it sacrifices).
Per-task evaluation
Headline AUROC alongside the eval-awareness-corrected and distribution-shift versions for each task. Reproducers run from a single notebook on a single Colab session.
| Task | n | AUROC [95% CI] | Eval-aware corrected | Dist-shift | ECE | FPR@99TPR | Latency | Reproducer |
|---|---|---|---|---|---|---|---|---|
Apollo Insider Trading Deception · deceptive | 150 | 0.978 [0.96, 0.99] | 0.910 | 0.840 | 0.040 | 0.012 | 2.1 ms | — |
Hypocrisy Gap Reasoning · unfaithful_cot | 180 | 0.800 [0.72, 0.87] | 0.710 | 0.650 | 0.090 | 0.100 | 2.1 ms | — |
Cross-model transfer (Pearson_CE)
Pearson correlation of paired ablation effects across models. ≥ 0.7 suggests the probe's direction is shared; 0.4–0.7 suggests partial transfer; < 0.4 suggests retraining per architecture.
| Source | Target | Pearson_CE | Transfer AUROC | Notes |
|---|---|---|---|---|
| Llama-3.3-70B | Qwen3.6-27B | 0.510 | 0.790 | |
| Llama-3.3-70B | Gemma-3-27B | 0.460 | 0.740 |
Architecture & artifact
Base model
- Family
- Llama
- Params
- 70B
- Architecture
- Dense transformer (instruct-tuned)
- Layers
- 80
- d_model
- 8192
- Weights license
- custom
Artifact
- Params
- 524,288
- Size
- 2.0 MB
- License
- Apache-2.0
- Released
- 2026-05-XX
- sha256
- dc4f2a8b6e5d…9d8c7b6a
openinterp probebench verify openinterp/deceptionguard-llama33-70b-l40.Reproduce
Three entry points. Pick one. Each path lands at the same numbers.
git clone https://github.com/OpenInterpretability/notebooks.git pip install openinterp openinterp probebench reproduce openinterp/deceptionguard-llama33-70b-l40
Honest scope
Limits derived from the evaluation data above. We think these are the honest constraints; revisions land as more reproducers do.
- Trained / evaluated on: Apollo Insider Trading, Hypocrisy Gap. Performance outside these tasks is unmeasured here.
- Eval-aware AUROC drop: −0.079 AUROC averaged across tasks (uncorrected − corrected for eval-awareness confound, arXiv:2509.13333).
- Distribution-shift drop: −0.144 AUROC averaged across tasks (in-distribution − long-context / OOD).
- Cross-model fit (mean Pearson_CE): 0.485 — values below 0.4 suggest retraining per architecture; values above 0.7 suggest the direction is shared.
- Probe attaches at: L40 · last_token of Llama-3.3-70B. Other layers / positions are out of scope unless re-trained.
Citations & endorsements
ProbeBench v0.0.1 · 5 probes registered · 11 evaluations · 7 transfer measurements. Schema and weights subject to revision.