---
license: apache-2.0
library_name: transformers
pipeline_tag: text-classification
base_model:
  - convaiinnovations/laya
  - answerdotai/ModernBERT-large
dataset: LocalLLaMA/typed-decisions
language:
- en
tags:
- decision-model
- system-one
- rlcd
- typed-decisions
- calibrated
- laya
---

# soft-decider-421m

An RLCD fine-tune of [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya)
(ModernBERT-large, 421M) on the [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions)
train split — a non-generative System-1 decision model: give it a state (JSON/text) plus typed
`choice` / `score` / `noul` questions, get calibrated probability distributions back in one forward
pass (~50 ms, GPU).

**What makes this checkpoint different from the published `laya-typed-decisions` is not raw
accuracy — it is calibration discipline.** The upstream checkpoint ships temperatures fitted on data
it had already trained on ([NandhaKishorM/laya#186](https://github.com/NandhaKishorM/laya/issues/186)),
so its confidence numbers are honestly marked "uncalibrated" by its own authors. Here the calibration
slice (400 decisions) is held out of training *before any batch is formed*, and both per-type and
per-option-count bucket temperatures are fitted on it.

## Measured — official test split (400 cases / 2,000 decisions)

Our harness reproduces `laya-typed-decisions` at **0.7664** against its published 0.766, so the ruler
is calibrated before it measures us.

| model | acc | soft acc | Brier | ECE ↓ | score MAE ↓ | flip ↓ | auto-decidable@5% | p50 ms |
|---|---|---|---|---|---|---|---|---|
| **soft-decider-421m** | **0.774** | **0.551** | 0.192 | **0.141** | **0.221** | 0.077 | 0.374 | 50 |
| laya-typed-decisions (this harness) | 0.766 | 0.500 | 0.213 | 0.214 | 0.243 | 0.065 | 0.378 | 50 |
| TypeSafe Jev 1.13.0 (published) | 0.727 | 0.580 | 0.148 | 0.144 | 0.391 | – | – | 710 |
| teacher self-agreement ceiling | 0.735 | – | – | – | – | – | – | – |

`flip` = share of `choice` decisions whose argmax changes when the option list is reshuffled.
`auto-decidable@5%` = share of traffic a confidence gate can automate at ≤5% running error —
the number that actually decides how much of an agent loop you can take off the LLM bill.

## Training recipe (fully reproducible, `scripts/`)

Single RTX 3090, bf16, full fine-tune (not LoRA), ~17 min: RLCD objective — zero-mean Gaussian
exploration on logits, reward = strictly proper scoring rule (log + spherical, w_sph=1.5, +RPS for
ordinal), REINFORCE with group-mean baseline (G=8), plus soft cross-entropy against the benchmark's
teacher distributions. 6 epochs, encoder LR 1.75e-5 / head LR 7e-5, cosine, held-out calibration
slice as above.

## Honest limits — read before deploying

- **Specialist.** Trained on and for the benchmark's four synthetic workflows. Outside them, expect
  base-checkpoint behaviour.
- **Out-of-domain failures are real and confident.** On 10 hand-written agent-loop scenarios
  (context compaction value, cost-rule batching, loop-breaking), this model *and* the published
  laya checkpoint were wrong on ~5, sometimes with high confidence — same-lineage blind spots, so
  ensembling the two does not fix it. Gate on confidence, route low-confidence decisions to an LLM
  or human tier, and consider a domain fine-tune on your own traces (`scripts/train_single.py --items ...`).
- **Never use it as the judge for rights / legal / money release gates** — those belong in
  deterministic rule code; a distribution is not a compliance decision.
- English states only. ~768 effective state tokens at the default 1024/256 budget; keep `choice`
  under ~20 options or raise `head_max_len`.
- Option-order flips are mitigated, not eliminated (7.7%): average over 2–3 orderings on hard calls.
- `action.act_probability` is inherited from upstream and carries no signal (#185). Use `confidence`.

## Use

```bash
pip install laya
```
```python
from huggingface_hub import snapshot_download
import laya

agent = laya.Agent(snapshot_download("winwinwinbb/soft-decider-421m"), device="cuda")
res = agent.predict(
    {"tool": "web_search", "result_summary": "docs page already used in step 5, step 9 of 20",
     "task_goal": "flag the render command"},
    {"keep": {"type": "noul", "instructions": "Is this tool result still needed for the task?"}},
)
# res["answers"]["keep"]["noul"] -> probability the result is still needed
```

A Jev-compatible server (same `POST /v1/systemone` shape as the TypeSafe API) is included at
`scripts/serve_soft.py`; official `typesafe-sdk` clients work against it by setting the base URL.

## Provenance

Built from `convaiinnovations/laya` (Apache-2.0, backbone `answerdotai/ModernBERT-large`) on the
`LocalLLaMA/typed-decisions` train split (Apache-2.0). Raw per-model metrics: `benchmark_report.json`.
Full training/eval scripts under `scripts/` — same harness produced every number on this card.