Whether the hosted Ekbasis API is up, how fast it answers, and how often it is right. Every number comes from machines outside our infrastructure calling the real API, and nothing is removed: bad hours and bad days stay.
ekbasis-27b-int4-v1.3A probe asks one small yes/no question every 10 minutes from GitHub Actions (the schedule can run late, so a window may hold fewer probes than 144 a day; the count is shown). Time is end to end from the caller, including the network. Measuring since 2026-10-12.
| Window | Answered | Probes | Median | p90 | Of which, model |
|---|---|---|---|---|---|
| 24 hours | 100% | 2 | 2.33 s | 3.05 s | 845 ms |
| 7 days | 100% | 2 | 2.33 s | 3.05 s | 845 ms |
| 30 days | 100% | 2 | 2.33 s | 3.05 s | 845 ms |
Every day the same fixed questions from the published test sets, none of them in the training data, are sent to the production API (100 per suite; the first run, made from our own machine before the daily job existed, used 300–400) by the open scorecard, which anyone can run against their own endpoint. A confident error is a wrong answer given with 90% confidence or more: the kind a guard cannot catch by asking when unsure. ECE is how far the stated confidence is from the observed accuracy (0 is perfect).
Latest run: 2026-10-13 01:04 UTC · Ekbasis-27B-INT4 v1.3
| Suite | Right | Confident errors / 100 | ECE | Median time | Questions |
|---|---|---|---|---|---|
| Git, known commandsreal git, command types seen in training | 98.0% | 2.0 | 0.019 | 715 ms | 100 |
| Git, unseen commandsreal git, command types never trained | 97.0% | 3.0 | 0.024 | 682 ms | 100 |
| Guard, unseen commandsthe larger guard set, held-out command types |
None recorded since measuring began.
Limits, what we keep and how model versions change: API policies. The API itself: OpenAPI.
| 89.0% |
| 11.0 |
| 0.107 |
| 637 ms |
| 100 |
| Worlds, unseen familiesworld families never trained | 86.0% | 4.0 | 0.070 | 515 ms | 100 |
| Agent intentagent steps from the published-attack replay | 100% | 0.0 | 0.019 | 1.24 s | 100 |
Chains (forecast, look, correct)
16/16 exact
6.7 looks per 100 actions · 0.46 s per action
Cost at the published price
$0.139 / 1,000 checks
463 input tokens per check, 0 generated
All 500 checks
p50 709 ms · p99 2.94 s
0 failed
| Run (UTC) | Git, known commands | Git, unseen commands | Guard, unseen commands | Worlds, unseen families | Agent intent | Median time | $ / 1k checks |
|---|---|---|---|---|---|---|---|
| 2026-10-13 01:04 | 98.0% · 2.0 | 97.0% · 3.0 | 89.0% · 11.0 | 86.0% · 4.0 | 100% · 0.0 | 709 ms | $0.139 |
| 2026-10-12 23:08 | 98.0% · 2.0 | 97.0% · 3.0 | 89.0% · 11.0 | 86.0% · 4.0 | 100% · 0.0 | 752 ms | $0.139 |
| 2026-10-12 22:47 | 97.8% · 2.3 | 95.8% · 3.8 | 83.8% · 14.0 | 83.5% · 4.8 | 100% · 0.0 | 839 ms | $0.125 |
Each cell: right · confident errors per 100. The questions are the same every day and the model answers them deterministically, so a number moves only when something changed: the model, the server or the gateway. The first run used more questions, so it differs.