OpenInterpretability
  • Ekbasis
  • Guide
  • Research
  • Lab
  • Tools
  • Notes
  • Registry
  • Manifesto
Download the modelDownload
OpenInterpretability

Ekbasis: know what an action will do before your agent runs it. From an independent lab for AI-agent safety. Open weights, Apache-2.0: download it and run it yourself.

Download the model
Ekbasis
  • Getting started
  • Download (Hugging Face)
  • For agents (agents.md)
  • Results and models
  • Client and CLI
  • Cookbook
The lab
  • About the lab
  • Open tools
  • Observatory
  • ProbeBench
  • InterpScore
  • Academy
Research
  • Manifesto
  • Roadmap
  • Papers & posts
  • Docs
Community
  • GitHub org
  • Notebooks
  • SDK (openinterp)
  • HuggingFace
© 2026 OpenInterpretability — Apache-2.0 for code, CC-BY 4.0 for docs.Built in public.
EkbasisNEW RELEASE · OPEN MODEL (APACHE-2.0) · OPEN DATA (PER-ROW LICENSES) · OPEN CODE

Ekbasis-Skopos-27B σκοπός

The next Ekbasis: one forward pass, trained further on what actions actually did.

Given the current state and an action, Skopos answers typed questions — will this lose work? will it fail? what will X be afterwards? — with a calibrated probability, in one forward pass, like Ekbasis-27B and with the same prompt, readout, API and client. It was trained further on executed shell, git, SQLite, PostgreSQL and Python scenarios and real git sequences.

  • The git guard raises fewer false alarms on command types held out of the git sandbox's training rows: 26 of 799 instead of 49 (−2.9 points, 95% CI [−4.1, −1.6], p < 0.001), and flags as many work-losing commands; on known types the differences are within noise.
  • World families it never saw: 83.4% instead of 81.8% on the 3,014 answers the actions change among 5,385 fresh items (+1.6 points, 95% CI [+0.7, +2.5]).
  • On a real repository the guard flagged 3/3 work-losing scenarios with 1/13 false alarm, and answered "will it fail?" right 17/17 (Ekbasis-27B: 3/3, 1/13, 16/17).

Every number on this page is read from the release's results files, with its pre-registration where there is one; the others are marked not pre-registered or added after review. the full report is linked below. Limits are at the bottom of the page.

Model · Hugging Face Data Code Evaluation report Pre-registrations Getting started

What changed from Ekbasis-27B

The pre-registered release protocol of the Ekbasis repository, the same items for both models, each served the same way. Rule: at most 1 point worse than the released model on each criterion, ECE at most +0.01 (PREREG_R4_RELEASE_READ.md).

git v3, known and held-out command types, all questions

Ekbasis-27B
96.62%
Skopos
96.94%
Met
yes

Never-trained worlds, answers the actions change: one prompt per question / read once

Ekbasis-27B
81.16% / 78.03%
Skopos
81.51% / 78.68%
Met
yes

ftest_family, answers the actions change (629 items)

Ekbasis-27B
81.88%
Skopos
80.29%
Met
no — see below

Calibration: ECE on git v3

Ekbasis-27B
0.024
Skopos
0.023
Met
yes

The guard, known command types: work-losing flagged / false alarms (of 863)

Ekbasis-27B
828 / 37
Skopos
837 / 27
Met
yes

The guard, held-out command types: work-losing flagged / false alarms (of 799)

Ekbasis-27B
767 / 49
Skopos
776 / 26
Met
yes

Real-repository git guard (T3c): flagged · false alarms · "fails?" right

Ekbasis-27B
3/3 · 1/13 · 16/17
Skopos
3/3 · 1/13 · 17/17
Met
yes
CriterionEkbasis-27BSkoposMet
git v3, known and held-out command types, all questions96.62%96.94%yes
Never-trained worlds, answers the actions change: one prompt per question / read once81.16% / 78.03%81.51% / 78.68%yes
ftest_family, answers the actions change (629 items)81.88%80.29%no — see below
Calibration: ECE on git v30.0240.023yes
The guard, known command types: work-losing flagged / false alarms (of 863)828 / 37837 / 27yes
The guard, held-out command types: work-losing flagged / false alarms (of 799)767 / 49776 / 26yes
Real-repository git guard (T3c): flagged · false alarms · "fails?" right3/3 · 1/13 · 16/173/3 · 1/13 · 17/17yes

The pre-registered release read missed one criterion: ftest_family, on its 629 items (−1.6 points, beyond the 1-point limit), so the release read as registered did not pass. A confirmation was then registered (after that result, before any fresh item existed): the same families on fresh items, by trainer readout; it did not confirm the loss:

"Held-out" command types are held out of the git-sandbox training rows; Skopos' shell and git factory rows include them, so for Skopos those rows are a weaker test than for Ekbasis-27B.

The confirmation on fresh items

5,385 fresh items of the never-trained families (cards, machines, timers), 3,014 of them answers the actions change (trainer readout).

83.4%

Skopos, answers the actions change

81.8%

Ekbasis-27B, same items

+1.6 [+0.7, +2.5]

difference, 95% CI

A git guard with fewer false alarms

"Will uncommitted work be lost?", flagged at P ≥ 0.2 (the guard's threshold), on the larger guard set (served), the same items for both models; paired differences (Skopos − Ekbasis-27B) with exact McNemar tests. The held-out command types (git rebase <branch>, git restore --source=HEAD~1 <file>) are held out of the git sandbox's training rows; Skopos' factory rows contain both, in other repositories.

26 / 799

false alarms, held-out types: −2.9 points [−4.1, −1.6], p < 0.001 (Ekbasis-27B 49)

27 / 863

false alarms, known types: −1.2 points [−2.3, 0.0], p = 0.08, not significant (Ekbasis-27B 37)

97.1%

work-losing flagged, held-out types: +1.1 points [−0.1, +2.5], p = 0.15, not significant (Ekbasis-27B 96.0%)

The 16 scenarios on a clone of pallets/itsdangerous that Ekbasis-27B's release ran (the repository has not changed since), the truth from running the commands in a copy. With an instruction planted in a file name, a branch name or the commit messages, no work-losing scenario dropped below the flag.

Ekbasis-Skopos-27B

Work-losing flagged
3/3
False alarms
1/13
"Will it fail?" right
17/17

Ekbasis-27B

Work-losing flagged
3/3
False alarms
1/13
"Will it fail?" right
16/17
ModelWork-losing flaggedFalse alarms"Will it fail?" right
Ekbasis-Skopos-27B3/31/1317/17
Ekbasis-27B3/31/1316/17

What the new training adds

Where the new training shows most: executed scenarios from the generators Skopos was trained on, with seeds it never saw — in-distribution for Skopos, new for Ekbasis-27B, so read these as what the training adds, not as generalization. These sets also chose between the candidate runs (by pre-registered rules), which adds selection on top; the independent evidence is the fresh never-seen families and the real agent turns below (trainer readout, paired 95% CI).

Executed factory scenarios (shell, SQL, Python, git)

Items
3,878
Ekbasis-27B
88.1%
Skopos
93.9%
Difference
+5.7 [+4.8, +6.7]

Real git sequences, 5–12 commands

Items
900
Ekbasis-27B
90.7%
Skopos
93.2%
Difference
+2.6 [+1.1, +3.9]

Same, validation twin

Items
1,800
Ekbasis-27B
91.0%
Skopos
94.6%
Difference
+3.6 [+2.6, +4.7]

Agent sessions (shell, read as a transcript)

Items
730
Ekbasis-27B
89.0%
Skopos
93.2%
Difference
+4.1 [+2.5, +5.8]
SuiteItemsEkbasis-27BSkoposDifference
Executed factory scenarios (shell, SQL, Python, git)3,87888.1%93.9%+5.7 [+4.8, +6.7]
Real git sequences, 5–12 commands90090.7%93.2%+2.6 [+1.1, +3.9]
Same, validation twin1,80091.0%94.6%+3.6 [+2.6, +4.7]
Agent sessions (shell, read as a transcript)73089.0%93.2%+4.1 [+2.5, +5.8]

Confident errors

The published confident-errors test, all 15,008 items as pre-registered (served bf16 vs Ekbasis-27B's published predictions). Wrong answers given with confidence ≥ 0.9 are what a guard would act on. In world families it never saw, Skopos is wrong slightly less often than Ekbasis-27B but more often with high confidence, and worse calibrated — a limit, below. Without the 5,381 items that overlap the lineage's training rows, the trained-family and held-out-question rows keep their direction (the evaluation report has both tables); no never-trained item overlaps.

Trained families

Items
7,172
Error rate: Ekbasis-27B → Skopos
4.27% → 3.64%
Confidently wrong per 100 (diff, 95% CI)
2.06 → 1.78 (−0.28 [−0.59, +0.03])
ECE (diff, 95% CI)
0.022 → 0.021 (−0.001 [−0.005, +0.003])

… answers the actions change

Items
2,940
Error rate: Ekbasis-27B → Skopos
7.24% → 6.02%
Confidently wrong per 100 (diff, 95% CI)
3.54 → 2.96 (−0.58 [−1.16, +0.03])
ECE (diff, 95% CI)
0.043 → 0.037 (−0.005 [−0.013, +0.001])

Held-out question types

Items
2,472
Error rate: Ekbasis-27B → Skopos
11.77% → 9.71%
Confidently wrong per 100 (diff, 95% CI)
4.00 → 3.64 (−0.36 [−1.21, +0.44])
ECE (diff, 95% CI)
0.055 → 0.054 (−0.001 [−0.014, +0.009])

… answers the actions change

Items
1,106
Error rate: Ekbasis-27B → Skopos
13.65% → 10.58%
Confidently wrong per 100 (diff, 95% CI)
4.88 → 3.80 (−1.08 [−2.44, +0.18])
ECE (diff, 95% CI)
0.067 → 0.053 (−0.014 [−0.032, +0.006])

Never-trained families

Items
5,364
Error rate: Ekbasis-27B → Skopos
14.93% → 14.26%
Confidently wrong per 100 (diff, 95% CI)
3.75 → 4.49 (+0.75 [+0.28, +1.23])
ECE (diff, 95% CI)
0.063 → 0.071 (+0.009 [+0.003, +0.015])

… answers the actions change

Items
2,700
Error rate: Ekbasis-27B → Skopos
14.67% → 14.85%
Confidently wrong per 100 (diff, 95% CI)
3.04 → 4.22 (+1.19 [+0.52, +1.89])
ECE (diff, 95% CI)
0.049 → 0.067 (+0.018 [+0.009, +0.028])
GroupItemsError rate: Ekbasis-27B → SkoposConfidently wrong per 100 (diff, 95% CI)ECE (diff, 95% CI)
Trained families7,1724.27% → 3.64%2.06 → 1.78 (−0.28 [−0.59, +0.03])0.022 → 0.021 (−0.001 [−0.005, +0.003])
… answers the actions change2,9407.24% → 6.02%3.54 → 2.96 (−0.58 [−1.16, +0.03])0.043 → 0.037 (−0.005 [−0.013, +0.001])
Held-out question types2,47211.77% → 9.71%4.00 → 3.64 (−0.36 [−1.21, +0.44])0.055 → 0.054 (−0.001 [−0.014, +0.009])
… answers the actions change1,10613.65% → 10.58%4.88 → 3.80 (−1.08 [−2.44, +0.18])0.067 → 0.053 (−0.014 [−0.032, +0.006])
Never-trained families5,36414.93% → 14.26%3.75 → 4.49 (+0.75 [+0.28, +1.23])0.063 → 0.071 (+0.009 [+0.003, +0.015])
… answers the actions change2,70014.67% → 14.85%3.04 → 4.22 (+1.19 [+0.52, +1.89])0.049 → 0.067 (+0.018 [+0.009, +0.028])

Confidence you can act on

Each answer can also come at a calibrated temperature (the additive `calibrated` field, client 0.1.41), fit on validation only (fval_git3 + m3val (validation only)). Not pre-registered: INT4 as served (Skopos on an RTX PRO 6000 with vLLM and serve.py; Ekbasis-27B on the hosted API's production server, an RTX 5090), on 2,482 items (git_long 900, guard_known 800, fac_val 782) from generators Skopos was trained on, with seeds it never saw; for Ekbasis-27B only the guard questions come from its training generators (git_long runs its sandbox at lengths it was not trained on; the factory is new to it). So this compares the two models on Skopos' home ground:

Accuracy

Ekbasis-27B
91.0%
Skopos
95.1%

Confidently wrong per 100, raw

Ekbasis-27B
6.3
Skopos
3.3

… at the calibrated temperature

Ekbasis-27B
3.9
Skopos
1.7

ECE, raw → calibrated

Ekbasis-27B
0.069 → 0.037
Skopos
0.035 → 0.015

Temperature

Ekbasis-27B
1.55
Skopos
1.67
INT4 as servedEkbasis-27BSkopos
Accuracy91.0%95.1%
Confidently wrong per 100, raw6.33.3
… at the calibrated temperature3.91.7
ECE, raw → calibrated0.069 → 0.0370.035 → 0.015
Temperature1.551.67

Each build ships its own temperature in decision_config.json (remove the key to turn the field off):

bf16

Calibrated temperature
1.62

INT4

Calibrated temperature
1.67

FP8

Calibrated temperature
1.64
BuildCalibrated temperature
bf161.62
INT41.67
FP81.64

Images

With the open builds the state can be text or an image, as in Ekbasis-27B (the hosted API reads text only). Skopos' last training was text only; on 395 new image items from world families it was trained on (only the image modality is untrained; the published drawings were regenerated, so pixels may differ; INT4 against INT4, served) its image ability is preserved by the pre-registered criterion.

79.8%

state given as an image (Ekbasis-27B 78.7%)

94.2%

reading the final state from the picture (Ekbasis-27B 91.9%)

91.9%

from text, same items

On real agent trajectories

455 turns of real agent trajectories (54 of them fail), from Qwen's AgentWorldBench: will the next action fail? Ranking (AUROC): the three Ekbasis models are within 0.010 of each other, inside their confidence intervals. Balanced accuracy at the fixed 0.5 threshold: Skopos is the lowest of the three; choose the threshold for your false-alarm budget, as the guard does. AgentWorld-35B-A3B (thinking off) left 59 answers unparsed, counted wrong; Skopos minus AgentWorld, balanced accuracy: +0.057 [−0.025, +0.136]. AgentWorld's training data may include these tasks.

Ekbasis-Skopos-27B

AUROC (95% CI)
0.843 [0.794, 0.889]
Balanced accuracy at 0.5 (95% CI)
0.696 [0.631, 0.752]

the first Skopos (unreleased)

AUROC (95% CI)
0.840 [0.789, 0.884]
Balanced accuracy at 0.5 (95% CI)
0.747 [0.687, 0.804]

Ekbasis-27B

AUROC (95% CI)
0.833 [0.783, 0.879]
Balanced accuracy at 0.5 (95% CI)
0.720 [0.653, 0.782]

AgentWorld-35B-A3B (thinking off, 87 s per answer)

AUROC (95% CI)
—
Balanced accuracy at 0.5 (95% CI)
0.640 [0.570, 0.701]
ModelAUROC (95% CI)Balanced accuracy at 0.5 (95% CI)
Ekbasis-Skopos-27B0.843 [0.794, 0.889]0.696 [0.631, 0.752]
the first Skopos (unreleased)0.840 [0.789, 0.884]0.747 [0.687, 0.804]
Ekbasis-27B0.833 [0.783, 0.879]0.720 [0.653, 0.782]
AgentWorld-35B-A3B (thinking off, 87 s per answer)—0.640 [0.570, 0.701]

Pre-launch evaluation

Before release, the checks of Ekbasis-27B's release evaluation that Skopos had not yet run were run on the same items and scripts, next to Ekbasis-27B's published results (PRELAUNCH_EVAL.md, sealed before its first request). The rule: a paired 95% bootstrap CI (2,000 resamples, seed 20261015) entirely on the worse side of 0 on a set of 100 items or more is a release blocker; smaller sets are counts, and a cell worse by 2 or more is flagged (PRELAUNCH_EVAL.md, Part A).

Release blocker, declared: on 120 short checks in paraphrased wording, Skopos got 113 right against Ekbasis-27B's 117 (−3.3 points, 95% CI [−6.7, −0.8]), so the pre-registered rule fired. The exact test on the same pairs does not reject (p = 0.125; 4 items differ, all against Skopos); Ekbasis-27B re-run on the same machine reproduced its answers item for item (240/240); in the original wording Skopos got 116 against 115, and over both wordings 229 against 232 of 240 (−1.25 points, [−3.3, +0.8]). It is consistent with the confident-error limit in never-trained world families. Released with it declared, by the owner's decision (2026-10-16).

No release blocker

  • P1, the 240-question git comparison: both halves: all 240 — 90.0% vs 90.8% (−0.8 [−3.8, +2.1])
  • P2, git v3 per question type: known command types: all questions — 98.6% vs 98.1% (+0.5 [−0.1, +1.0]); never-seen command types: all questions — 95.4% vs 95.3% (+0.2 [−0.6, +1.0])
  • P3, long chains: 32 measures, no cell worse by 2 or more
  • P4, predict, observe, correct on long chains: 200 more chains, check0.9 (published table): exact at the end — 99.5% vs 99.0% (+0.5 [−1.0, +2.5]); 200 more chains, conf0.5 (run, not in the published table): exact at the end — 99.5% vs 98.5% (+1.0 [−1.0, +3.0])
  • P5, speed and accuracy of 80 forecasts: one prompt per question: accuracy — 78 vs 79 of 80; state read once per step: accuracy — 75 vs 72 of 80
  • P6, Portuguese: English, all: changed answers right — 78 vs 75 of 90; Portuguese, all: changed answers right — 75 vs 74 of 90
  • P7, chains that start from an image: final answer from the image, all — 60 vs 60 of 60
  • P9, planning by search: both sets: plans that work — 99.2% vs 99.4% (−0.3 [−0.8, 0.0])
  • P10, real use on real repositories: work-losing scenarios caught — 28 vs 28 of 28; false alarms — 0 vs 0 of 44; 'will it fail?' right — 95 vs 92 of 96
  • P10 on the INT4 build (against Ekbasis-27B's published bf16 run): work-losing scenarios caught — 28 vs 28 of 28; false alarms — 0 vs 0 of 44; 'will it fail?' right — 95 vs 92 of 96
  • P12, intent_check replay on the INT4 build, against the hosted Ekbasis-27B-INT4's published run: real successful attacks stopped 2,092 of 2,092, as Ekbasis-27B; benign runs with a false alarm 7 against 17 of 1,067, fewer (−0.94 points [−1.50, −0.37]; first replay 5 against 13, exact McNemar p = 0.0078). 6 replayed runs, 3 per replay, are a benchmark artifact the study found (banking user_task_15 x injection_task_4, benchmark v1: the user's own request names the attacker's IBAN): left out of both models' denominators, as in the study; neither model stops them

Flagged (counts; Skopos worse by 2 or more)

  • P1 seen command types: lost — 37 vs 39 of 40
  • P1 seen command types: state (changed) — 37 vs 39 of 40
  • P4 rules, conf0.5: exact at the end — 38 vs 40 of 40
  • P4 rules, until0.5: exact at the end — 38 vs 40 of 40
  • P4 rules, check0.5: exact at the end — 38 vs 40 of 40
  • P6 English, toggles: changed answers right — 22 vs 24 of 30
  • P7 start state read exactly, all — 49 vs 51 of 60
  • P7 start state read exactly, jugs — 20 vs 22 of 30

Intervals that touch 0 (not blockers)

  • P2 known command types: will the command fail? (fails) — 97.1% vs 97.8% (−0.7 [−2.2, 0.0])
  • P2 never-seen command types: is an operation left in progress? (in_progress) — 95.4% vs 97.4% (−2.0 [−4.0, 0.0])
  • P9 both sets: plans that work — 99.2% vs 99.4% (−0.3 [−0.8, 0.0])
  • P9 goals from 8-12-action plans: plans that work — 98.3% vs 98.9% (−0.6 [−1.7, 0.0])

Better than Ekbasis-27B beyond the interval

  • P2 known command types: question type clean (all its questions) — 98.6% vs 95.9% (+2.7 [+0.7, +5.5])
  • P4 200 more chains, check0.9 (published table): looks per 100 actions — 10.18 vs 16.94 (−6.76 [−7.78, −5.72])
  • P4 200 more chains, conf0.5 (run, not in the published table): looks per 100 actions — 5.50 vs 8.38 (−2.88 [−3.45, −2.30])

Not run as published: P12 preflight on 141 plans (plans' snapshots lost).

Still running at release; their results are appended to the evaluation report on completion: P11, GPU memory of each build.

Pick the build that fits your machine

Every build is evaluated on the same release protocol against the bf16 model, served; the quantization gate was pre-registered. An MLX build for Apple Silicon follows after its test on Apple hardware.

Ekbasis-Skopos-27B

Size
54.7 GB
Runs on
GPUs with 80 GB
Against bf16
the reference (below)

Ekbasis-Skopos-27B-INT4

Size
18.6 GB
Runs on
GPUs with 32 GB
Against bf16
quantization gate passed (same answer as bf16 on 97.4%); git v3 96.8% (bf16 96.9%); guard false alarms 25 · 25 (bf16 27 · 26); fresh never-trained families +1.7 (this build served, against Ekbasis-27B bf16 by trainer readout)

Ekbasis-Skopos-27B-FP8

Size
30.4 GB
Runs on
GPUs with 48 GB
Against bf16
quantization gate passed (same answer as bf16 on 98.6%); git v3 96.9% (bf16 96.9%); guard false alarms 27 · 26 (bf16 27 · 26); fresh never-trained families +1.4 (this build served, against Ekbasis-27B bf16 by trainer readout)
BuildSizeRuns onAgainst bf16
Ekbasis-Skopos-27B54.7 GBGPUs with 80 GBthe reference (below)
Ekbasis-Skopos-27B-INT418.6 GBGPUs with 32 GBquantization gate passed (same answer as bf16 on 97.4%); git v3 96.8% (bf16 96.9%); guard false alarms 25 · 25 (bf16 27 · 26); fresh never-trained families +1.7 (this build served, against Ekbasis-27B bf16 by trainer readout)
Ekbasis-Skopos-27B-FP830.4 GBGPUs with 48 GBquantization gate passed (same answer as bf16 on 98.6%); git v3 96.9% (bf16 96.9%); guard false alarms 27 · 26 (bf16 27 · 26); fresh never-trained families +1.4 (this build served, against Ekbasis-27B bf16 by trainer readout)

Upgrade

Self-hosted: download a build and point the client at it — the API, the client, the git and shell guards, the Claude Code hooks, check_plan, monitor() and simulate work unchanged.

Hosted API: it serves Ekbasis-Skopos-27B-INT4 since 2026-10-16 14:29 UTC (it served Ekbasis-27B-INT4 until then). The switch came on release day, with less notice than the 14 days our API policy promises; Ekbasis-27B's weights stay public for anyone who needs the previous model. Every answer names its model in a response header. The hosted API reads text only; the open builds served with serve_vllm.sh also read images.

Certified mode of ekbasis.verify (thresholds certified on Ekbasis-27B): checked with Skopos on 1,400 fresh scenarios — INT4: 7 errors among 927 answers accepted without verifying (weighted 0.74%, 95% upper bound 1.41%; target at most 2%), 22.0% verified; bf16: 6 among 935 (bound 1.26%). On the same scenarios Ekbasis-27B bf16 made 5 among 886 (weighted 0.47%, 25.4% verified): Skopos accepts more and errs slightly more often. Per rule node the check has little power (43 nodes, median 16 accepted answers; the weakest 2 errors in 12, p = 0.023 uncorrected, not significant after Bonferroni). The certificate carries over; a check, not a new certificate.

hf download caiovicentino1/Ekbasis-Skopos-27B-INT4 --local-dir Ekbasis-Skopos-27B-INT4
bash Ekbasis-Skopos-27B-INT4/serve_vllm.sh Ekbasis-Skopos-27B-INT4 8001
python Ekbasis-Skopos-27B-INT4/serve.py --model Ekbasis-Skopos-27B-INT4 --vllm-url http://127.0.0.1:8001 --port 8000
pip install -U ekbasis && EKBASIS_URL=http://127.0.0.1:8000 ekbasis git-check -- "git reset --hard"

Scorecard for the hosted API

The scorecard of the hosted API's change of 2026-10-16 14:29 UTC, posted the same day (our API policy asks for 14 days; this switch came on release day). It compares Ekbasis-Skopos-27B-INT4 (what the hosted API serves since 2026-10-16) against Ekbasis-27B-INT4 (what it served until then). Accuracy: served, the release protocol's sets (the same items for both).

git v3, known and held-out command types, all questions

Ekbasis-Skopos-27B-INT4
96.78%
Ekbasis-27B-INT4
96.43%

never-trained worlds, answers the actions change, one prompt per question

Ekbasis-Skopos-27B-INT4
81.51%
Ekbasis-27B-INT4
80.98%

never-trained worlds, answers the actions change, read once

Ekbasis-Skopos-27B-INT4
77.91%
Ekbasis-27B-INT4
77.08%

ftest_family, answers the actions change (629)

Ekbasis-Skopos-27B-INT4
80.45%
Ekbasis-27B-INT4
80.92%

ECE on git v3 (10 bins)

Ekbasis-Skopos-27B-INT4
0.025
Ekbasis-27B-INT4
0.025

the guard at 0.2, larger set: work-losing flagged / false alarms, known (of 863) and held-out (of 799) types

Ekbasis-Skopos-27B-INT4
836 / 25 · 778 / 25
Ekbasis-27B-INT4
not measured on this set (Ekbasis-27B bf16: 828 / 37 · 767 / 49)

never-trained families, fresh confirmation (5,385 items; against Ekbasis-27B bf16, trainer readout)

Ekbasis-Skopos-27B-INT4
+1.7 points
Ekbasis-27B-INT4
reference
SuiteEkbasis-Skopos-27B-INT4Ekbasis-27B-INT4
git v3, known and held-out command types, all questions96.78%96.43%
never-trained worlds, answers the actions change, one prompt per question81.51%80.98%
never-trained worlds, answers the actions change, read once77.91%77.08%
ftest_family, answers the actions change (629)80.45%80.92%
ECE on git v3 (10 bins)0.0250.025
the guard at 0.2, larger set: work-losing flagged / false alarms, known (of 863) and held-out (of 799) types836 / 25 · 778 / 25not measured on this set (Ekbasis-27B bf16: 828 / 37 · 767 / 49)
never-trained families, fresh confirmation (5,385 items; against Ekbasis-27B bf16, trainer readout)+1.7 pointsreference

The regression a hosted user should know about — never-trained world families of the published confident-errors study: wrong at confidence ≥ 0.9 per 100, measured on the INT4 (served), both builds (PREREG_ADDENDUM A7): raw answers, Skopos 4.75 against Ekbasis-27B 3.65 on 5,364 items (+1.10 [+0.65, +1.57]). Reading the calibrated field (T 1.67 and 1.55) cuts them to 1.62 and 1.10: Skopos calibrated is below Ekbasis-27B raw, and calibrated against calibrated Skopos still has +0.52 per 100 [+0.26, +0.78]; calibrated ECE is level (0.021 against 0.022). A7 on bf16 not run (the evaluation machine was retired).

Confident errors and calibration on 2,482 items, served through the System One API, both INT4 (from generators Skopos was trained on: its home ground).

Accuracy

Skopos INT4
95.1%
Ekbasis-27B INT4
91.0%

Wrong at confidence ≥ 0.9, per 100: raw

Skopos INT4
3.3
Ekbasis-27B INT4
6.3

… at the calibrated temperature

Skopos INT4
1.7
Ekbasis-27B INT4
3.9

ECE: raw → calibrated

Skopos INT4
0.035 → 0.015
Ekbasis-27B INT4
0.069 → 0.037

Calibrated temperature

Skopos INT4
1.67
Ekbasis-27B INT4
1.55
Skopos INT4Ekbasis-27B INT4
Accuracy95.1%91.0%
Wrong at confidence ≥ 0.9, per 100: raw3.36.3
… at the calibrated temperature1.73.9
ECE: raw → calibrated0.035 → 0.0150.069 → 0.037
Calibrated temperature1.671.55

Latency and cost on 200 fixed items, one RTX PRO 6000, vLLM + serve.py, nothing else on the GPU (the hosted API runs on an RTX 5090). Cost on these fixed items (about 300 input tokens each); a check's cost scales with its input tokens.

One at a time: median / p90

Ekbasis-Skopos-27B-INT4
0.064 s / 0.068 s
Ekbasis-27B-INT4
0.064 s / 0.068 s

16 in parallel

Ekbasis-Skopos-27B-INT4
19.0 requests/s
Ekbasis-27B-INT4
19.0 requests/s

Input tokens per request

Ekbasis-Skopos-27B-INT4
302
Ekbasis-27B-INT4
302

Cost per 1,000 checks (at $0.30 per 1M input tokens)

Ekbasis-Skopos-27B-INT4
$0.090
Ekbasis-27B-INT4
$0.090
Ekbasis-Skopos-27B-INT4Ekbasis-27B-INT4
One at a time: median / p900.064 s / 0.068 s0.064 s / 0.068 s
16 in parallel19.0 requests/s19.0 requests/s
Input tokens per request302302
Cost per 1,000 checks (at $0.30 per 1M input tokens)$0.090$0.090

Against a general-purpose calibrated evaluator

A general-purpose calibrated evaluator, Jev (typesafe-ai/jev on the Vercel AI Gateway, zero data retention), on the same items with the same state text: it, too, answers typed questions with probabilities in one pass. Measured 2026-10-16, not pre-registered; two prompt variants (Ekbasis' question and one written in Jev's documented style), the headline variant chosen per set by a rule fixed before scoring. 14,976 calls cost US$ 0.34.

Ekbasis-Skopos-27B and Jev on the same questions: knows what a git command will do, 98.4 vs 90.4 (known types), 95.3 vs 89.5 (types kept out of the git training set), 83.8 vs 74.8 (rule-based worlds); warns before you lose work at the same false-alarm rate, 97.3 vs 45.3 and 98.9 vs 43.7; false alarms on safe commands, 2.9 vs 84.9 and 3.1 vs 93.6. Not shown: Jev is rarely confidently wrong and costs about 4x less per check; the full table is below.

git v3, 3,105 questions: accuracy

Jev
90.0%
Ekbasis-27B (bf16, published)
96.6%
Ekbasis-Skopos-27B-INT4
96.8%

git guard, known types (1,726): will it lose work?

Jev
AUROC 0.879; at 0.2: 99.7% caught, 84.9% false alarms; at Ekbasis-27B's false-alarm rate: 45.3% caught
Ekbasis-27B (bf16, published)
AUROC 0.982; at 0.2: 95.9% caught, 4.3% false alarms; at Ekbasis-27B's false-alarm rate: 95.9% caught
Ekbasis-Skopos-27B-INT4
AUROC 0.992; at 0.2: 96.9% caught, 2.9% false alarms; at Ekbasis-27B's false-alarm rate: 97.3% caught

git guard, held-out types (1,598)

Jev
AUROC 0.871; at 0.2: 100.0% caught, 93.6% false alarms; at Ekbasis-27B's false-alarm rate: 43.7% caught
Ekbasis-27B (bf16, published)
AUROC 0.985; at 0.2: 96.0% caught, 6.1% false alarms; at Ekbasis-27B's false-alarm rate: 96.2% caught
Ekbasis-Skopos-27B-INT4
AUROC 0.993; at 0.2: 97.4% caught, 3.1% false alarms; at Ekbasis-27B's false-alarm rate: 98.9% caught
Set (same items)JevEkbasis-27B (bf16, published)Ekbasis-Skopos-27B-INT4
git v3, 3,105 questions: accuracy90.0%96.6%96.8%
git guard, known types (1,726): will it lose work?AUROC 0.879; at 0.2: 99.7% caught, 84.9% false alarms; at Ekbasis-27B's false-alarm rate: 45.3% caughtAUROC 0.982; at 0.2: 95.9% caught, 4.3% false alarms; at Ekbasis-27B's false-alarm rate: 95.9% caughtAUROC 0.992; at 0.2: 96.9% caught, 2.9% false alarms; at Ekbasis-27B's false-alarm rate: 97.3% caught
git guard, held-out types (1,598)AUROC 0.871; at 0.2: 100.0% caught, 93.6% false alarms; at Ekbasis-27B's false-alarm rate: 43.7% caughtAUROC 0.985; at 0.2: 96.0% caught, 6.1% false alarms; at Ekbasis-27B's false-alarm rate: 96.2% caughtAUROC 0.993; at 0.2: 97.4% caught, 3.1% false alarms; at Ekbasis-27B's false-alarm rate: 98.9% caught

git3_known

Jev
91.5% (0.50 confidently wrong per 100)
Ekbasis-27B-INT4
97.8% (2.25 confidently wrong per 100)
Ekbasis-Skopos-27B-INT4
98.2% (1.25 confidently wrong per 100)

git3_held

Jev
92.0% (0.25 confidently wrong per 100)
Ekbasis-27B-INT4
95.5% (3.75 confidently wrong per 100)
Ekbasis-Skopos-27B-INT4
95.2% (3.75 confidently wrong per 100)

guard_held

Jev
74.5% (0.00 confidently wrong per 100)
Ekbasis-27B-INT4
83.2% (bf16) (14.50 confidently wrong per 100)
Ekbasis-Skopos-27B-INT4
86.8% (10.75 confidently wrong per 100)

worlds

Jev
74.8% (0.75 confidently wrong per 100)
Ekbasis-27B-INT4
83.5% (4.75 confidently wrong per 100)
Ekbasis-Skopos-27B-INT4
83.8% (7.50 confidently wrong per 100)

intent

Jev
100.0% (0.00 confidently wrong per 100)
Ekbasis-27B-INT4
not published per item
Ekbasis-Skopos-27B-INT4
not published per item
Hosted scorecard suiteJevEkbasis-27B-INT4Ekbasis-Skopos-27B-INT4
git3_known91.5% (0.50 confidently wrong per 100)97.8% (2.25 confidently wrong per 100)98.2% (1.25 confidently wrong per 100)
git3_held92.0% (0.25 confidently wrong per 100)95.5% (3.75 confidently wrong per 100)95.2% (3.75 confidently wrong per 100)
guard_held74.5% (0.00 confidently wrong per 100)83.2% (bf16) (14.50 confidently wrong per 100)86.8% (10.75 confidently wrong per 100)
worlds74.8% (0.75 confidently wrong per 100)83.5% (4.75 confidently wrong per 100)83.8% (7.50 confidently wrong per 100)
intent100.0% (0.00 confidently wrong per 100)not published per itemnot published per item

Reading: on questions about the repository's state Jev comes within 6–7 points; as a guard it does not separate work-losing commands from safe ones (its probabilities sit near the middle), so at the guard's threshold it flags nearly everything. In its favour: it is about four times cheaper per check, and it is almost never confidently wrong — fewer confident errors than Ekbasis-27B and Skopos on every suite where their per-item answers are published. Method, per-item answers and a script that recomputes every number: results_skopos/jev_comparison in the code repository.

Training data, code and checks — open, with the exceptions listed

Lineage: Qwen3.8-27B → Eikos-27B → Ekbasis-27B → the first Skopos (v49f_w50, not released) → Ekbasis-Skopos-27B.

The first Skopos: Ekbasis-27B's adapter (rank 128) trained further for 900 steps, then the exact interpolation halfway between it and Ekbasis-27B's own adapter (WiSE, λ 0.5), merged; an intermediate checkpoint that was not released: its weights are archived privately, and the published code and data retrain it approximately. Then a new LoRA (rank 64, all linear layers of the language model, 1,200 steps, cross-entropy on the answer letter plus the JEPA-style hindsight term) trained on the first Skopos and merged into its weights.

λ 0.5 was chosen by the selection rule of its plan (PLAN_v49b), on a validation set of world-family questions (trained, held-out question types and never-trained families): the un-interpolated checkpoint failed its accuracy and calibration criteria (mean accuracy over the groups 88.19% against a floor of 88.85%; ECE 0.061 against Ekbasis-27B's 0.048, tolerance +0.01), λ 0.75 failed the accuracy criterion (88.79%), and λ 0.5 passed both (89.30%, ECE 0.051). The last stage's selection (R1 → R2 → R4) had no confident-error or world-family calibration criterion, and the released model shows more confident errors in never-trained families (Limits).

Its training data: shell, git, SQLite, PostgreSQL and Python scenarios generated by our factories and executed in throwaway sandboxes; the last stage used the factory regenerated with fresh seeds; real git sequences of 4–16 commands executed in throwaway repositories, and guard-style git items with their executed labels; plan-level items in synthetic rule-based worlds, with exact answers from our simulators; a replay of Ekbasis-27B's training mixture (simulated worlds, git, and the eikos-decisions replay rows).

Not in the training data: Vikunja rows built during development were all dropped by the overlap firewall; WebWorld transitions were excluded by rule.

Open: the weights; the training rows of every stage (except 7,423 replay rows that are not part of the published eikos-decisions (as in ekbasis-data) and 31 more rows withheld by its privacy scan, 26 of them CWM rows (all listed by id)); the trainers and recipes as they ran; the generators that computed the rows; every evaluation with its per-item predictions (except the quantized builds' answers on the fresh confirmation, kept as summaries); the candidate runs that were not released and why; and the pre-registrations.

shio-bash-26-1

Rows
600
License
CC BY 4.0
How it was used
command lines re-executed in our sandbox; labels unused

swe-rebench-v2-cwm

Rows
2,000
License
CC BY 4.0 + each source repository's license
How it was used
values recorded by that dataset's tracer, not re-executed
External sourceRowsLicenseHow it was used
shio-bash-26-1600CC BY 4.0command lines re-executed in our sandbox; labels unused
swe-rebench-v2-cwm2,000CC BY 4.0 + each source repository's licensevalues recorded by that dataset's tracer, not re-executed

Contamination audit of every evaluation set against every training row of the lineage: identical prompts (also against the hindsight prompt), and near-duplicates at word 8-gram Jaccard >= 0.5 against one training text after removing each set's own layout (Ekbasis-27B's audit rule, exact best match). Not covered: Eikos-27B's own decision data, Qwen's pretraining, T3c (its scenarios are not dataset items). Items flagged: multi_test_trainfam (reported only) 291 of 600; fac_val 105 of 3,983; fhorizon 547 of 1,375; image (new seeds) 5 of 400; confident-errors set 5,381 of 15,008. Numbers that use those sets are reported without the flagged items as well.

Honest scope

  • Everything in Ekbasis-27B's limits applies, including confident errors in the world families it was trained on.
  • More confident errors in unfamiliar worlds: on the 5,364 items of never-trained world families of the published confident-errors set (served bf16), Skopos is wrong slightly less often than Ekbasis-27B (14.26% against 14.93%), but more often with confidence 0.9 or more: 4.49 against 3.75 wrong answers per 100 (+0.75 [+0.28, +1.23]; on the answers the actions change +1.19 [+0.52, +1.89]), and it is worse calibrated there (ECE 0.071 against 0.063, +0.009 [+0.003, +0.015]). The selection of this version had no criterion on confident errors in these families (Training). In unfamiliar domains, let it look when unsure (simulate) and act on a confident answer only at a stricter threshold than in familiar ones. In never-trained world families, reading calibrated cuts Skopos' confidently wrong answers by about two-thirds (INT4: 4.75 → 1.62 per 100), to fewer than Ekbasis-27B's raw answers (3.65). Calibrated against calibrated, Skopos still has about 0.5 more per 100 answers there than Ekbasis-27B (1.62 vs 1.10, +0.52 [+0.26, +0.78]); ECE is level (0.021 vs 0.022). Measured on the INT4 builds as PREREG_ADDENDUM.md A7 pre-registered; A7 on bf16 was not run (the evaluation machine was retired).
  • Paraphrased wording (pre-launch P8): on 120 short checks in paraphrased wording, Skopos is right on 113 against Ekbasis-27B's 117 (−3.3 points [−6.7, −0.8]; exact McNemar p = 0.125), 2 of the 4 items it alone gets wrong in a never-trained family: the release blocker declared in the Pre-launch evaluation.
  • In-distribution and selected gains: the largest gains (executed factory scenarios, real git sequences, agent sessions) are on sets built by the same generators as its training rows, and three of them (fac_val, git_long, git_long_val) are the sets whose pre-registered rules chose this version over its siblings (R1 → R2 → R4): selection adds to the in-distribution advantage. The independent evidence is smaller: the fresh never-trained families (+1.6 points, trainer readout) and the neutral agent turns (AUROC level with Ekbasis-27B). The never-trained families are never in training rows; the first Skopos' λ was chosen on a validation set that includes these families (cards, machines, tally, timers and track), and each stage's checkpoint rule was the best step on fval_git3, which holds Track and Tally items and is part of the calibration fit set (both stages kept their last step: the first Skopos' training run, v49f, saved only step 900, and Skopos' best was its last, step 1,200; training/skopos/README.md).
  • "Held-out" git command types: the release protocol's held-out types (git rebase <branch> and git restore --source=HEAD~1 <file>) are held out of the git sandbox's training rows, but Skopos' shell and git factory rows contain both, in other repositories (the overlap audit's note): read the held-out rows as a weaker test for Skopos than for Ekbasis-27B.
  • Long chains in written-rule worlds: answering in one pass, its accuracy falls with the number of actions (87.9% over 1–20 actions on 828 items, the overlap-free part of fhorizon, from the generator of its plan-level training rows: in-distribution); a model that simulates step by step does better there at far higher cost. Use simulate (look when unsure) for long chains.
  • Images: the state can be text or an image, as in Ekbasis-27B. This version's added training was text only; on 395 image items from trained world families (lamps and containers, new seeds; overlap-free; INT4 against INT4, served), its image ability is preserved by the pre-registered criterion (PREREG_IMAGE.md: at most 2 points lower): 79.8% against 78.7% when the state is an image (+1.0 [−2.0, +4.1]), and reading the final state from a picture 94.2% against 91.9% (+2.3 [+0.5, +4.3]). Only the modality is new to its added training, not the worlds. Predicting from a picture stays weaker than from text (91.9% on the same items). The drawings published in the dataset were regenerated from the same states; their pixels may differ from the ones evaluated.