Eight alignment-relevant categories. Seven axes weighted into a single ProbeScore. Every entry ships with a Colab reproducer, an SHA-256-hashed test set, and Pearson_CE cross-model transfer numbers. We are publishing the slot we want our own probes to compete in.
New: every probe here is now agent-callable via openinterp-mcp — any Claude Code / Cursor / Cline session can score or audit a probe against your own Colab compute.
Probing the residual stream is one of the most reproducible interpretability primitives we have — and one of the easiest to mis-report. ProbeScore composes the three corrections most likely to surface the difference between a real probe and a confound.
Published probes hit 0.96+ on Apollo deception suites, but break under long-context distribution shift (DeepMind 2601.11516) and inflate when test prompts trigger eval-awareness (arXiv 2509.13333). ProbeScore composites both corrections so the headline number is harder to oversell.
Naive cosine alignment misses 38% of causal direction across crosscoders. Pearson_CE, the paired-ablation correlation between source and target models, is the honest transfer metric. Every cross-model entry on this page ships one.
No axis weighted above 30%. License score capped for closed-weight probes. Eval-awareness component dampens when calibration is missing. Hard to game by optimizing one number, easy to PR if you disagree with the weights.
Sorted by ProbeScore across all 8 categories. Click any row for the full Probe DNA card — eval table, Pearson_CE matrix, license, artifact hash.
| # | Probe | Category | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|---|
| 1 | EvalAwarenessGuard Caio Vicentino · OpenInterp | Eval-Aware | Qwen3.6-27B · L40 | 0.930 | 1.0 | Apache-2.0 | 0.848 | HF |
| 2 | DeceptionGuard Caio Vicentino (re-impl) / Apollo Research (method) · OpenInterp | Deception | Llama-3.3-70B-Instruct · L40 | 0.978 | 2.1 | Apache-2.0 | 0.785 | HF |
| 3 | FabricationGuard Caio Vicentino · OpenInterp | Hallucination | Qwen3.6-27B · L31 | 0.903 | 1.0 | Apache-2.0 | 0.662 | HF |
| 4 | RewardHackGuard Caio Vicentino · OpenInterp | Reward-Hack | Qwen3.6-27B · L31 | 0.650 | 1.8 | Apache-2.0 | 0.645 | HF |
| 5 | ReasonGuard Caio Vicentino · OpenInterp | Reasoning | Qwen3.6-27B · L55 | 0.908 | 1.0 | Apache-2.0 | 0.626 | HF |
Each alignment-relevant slot has its own table, its own held-out test set, and its own empty rows for probes that haven't been submitted yet.
Factual fabrication, entity recall failures.
| # | Probe | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|
| 1 | FabricationGuard Caio Vicentino · OpenInterp | Qwen3.6-27B · L31 | 0.903 | 1.0 | Apache-2.0 | 0.662 | HF |
Chain-of-thought faithfulness, hypocrisy gap.
| # | Probe | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|
| 1 | ReasonGuard Caio Vicentino · OpenInterp | Qwen3.6-27B · L55 | 0.908 | 1.0 | Apache-2.0 | 0.626 | HF |
Strategic dishonesty, sleeper agents, alignment faking.
| # | Probe | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|
| 1 | DeceptionGuard Caio Vicentino (re-impl) / Apollo Research (method) · OpenInterp | Llama-3.3-70B-Instruct · L40 | 0.978 | 2.1 | Apache-2.0 | 0.785 | HF |
Capability hiding under evaluation.
Model knows-it-is-being-tested confound.
| # | Probe | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|
| 1 | EvalAwarenessGuard Caio Vicentino · OpenInterp | Qwen3.6-27B · L40 | 0.930 | 1.0 | Apache-2.0 | 0.848 | HF |
RL-induced misalignment generalization.
| # | Probe | Model | AUROC | Latency (ms) | License | ProbeScore | Artifact |
|---|---|---|---|---|---|---|---|
| 1 | RewardHackGuard Caio Vicentino · OpenInterp | Qwen3.6-27B · L31 | 0.650 | 1.8 | Apache-2.0 | 0.645 | HF |
EU AI Act Article 5 — persuasion / subliminal.
Over/under-refusal calibration; jailbreak proxy.
Seven axes, each in [0, 1], weighted to sum to 1.0. We report every component alongside the headline score so you can re-aggregate with your own weights.
Where these probes apply in practice. CZI Biohub committed $500M to AI-biology on Apr 29 2026 — medical AI is the most urgent vertical for fabrication detection.
Two paths — pick whichever fits your workflow. Both end in the same place: a sklearn-compatible artifact + hashed test set + Colab reproducer registered on the leaderboard.
0.0.1openinterp-mcp > publish_probe() auto-opens the PR + mints a Zenodo DOI + creates an HF datasetpredict_proba(X) -> (n, 2) + StandardScaler + metadata blockTwo ways to score: the Python SDK (deterministic, no GPU needed for evaluation metadata) or the MCP server (agent runs the full eval suite against your own Colab compute, returns a ProbeScore card identical to the leaderboard).
pip install -U openinterp
from openinterp import probebench
probe = probebench.load("openinterp/fabricationguard-qwen36-27b-l31-v2")
print(probebench.score(probe))# in Claude Code / Cursor / Cline session: /colab-attach https://abc123.ngrok-free.app > score probebench:openinterp/fabricationguard-qwen36-27b-l31-v2 on tasks: haluval-qa, simpleqa