Add HF-format export at step 740,000 (72.745B tokens), model card and training charts

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+loss_curve.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,5 +1,155 @@
-this is just for transfering checkpoints between instances, why are you here
+---
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- causal-lm
+- language-model
+- base-model
+- pretrained-from-scratch
+- small-language-model
+datasets:
+- HuggingFaceTB/smollm-corpus
+- mlfoundations/dclm-baseline-1.0
+- HuggingFaceTB/finemath
+- nvidia/OpenMathInstruct-2
+- HuggingFaceTB/smoltalk
+---
 
+# cagliostro-v3
 
+A 146M parameter decoder-only language model pretrained from scratch on 75B tokens of open web, synthetic textbook and mathematics data. It is the third model in the cagliostro line and the first to clear an Index of 26 on the [Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard) metric.
 
-*PLEASE STOP LIKING THIS ONE*
+> **This is a mid-training checkpoint.** The run is at **72.745B of 75B tokens** and **80% of the way through** its learning rate cooldown. Roughly 2.3B tokens remain. The numbers below are measured on the weights in this repository as they stand today, not on a finished model. They should improve slightly when the run completes, and this card will be updated when it does.
+
+## Results
+
+Zero-shot, measured with lm-evaluation-harness and the leaderboard's own ArithMark-3 script, on the exported float32 weights in this repository.
+
+| Benchmark | Metric | Score |
+|---|---|---:|
+| HellaSwag | acc_norm | 42.42 |
+| ARC-Easy | acc_norm | 54.17 |
+| ARC-Challenge | acc_norm | 27.99 |
+| PIQA | acc_norm | 67.46 |
+| ArithMark-3 | acc_norm | 43.80 |
+| **Open SLM Index** | | **26.27** |
+
+The Index is the leaderboard's own formula, `(N(HellaSwag,25) + N(CombinedARC,25) + N(PIQA,50) + 0.65*N(ArithMark,25)) / 3.65` where `N(v,c) = 100(v-c)/(100-c)` and CombinedARC is the mean of ARC-Easy and ARC-Challenge.
+
+For context against other models at this scale, using the leaderboard's published figures:
+
+| Model | Params | Tokens | Index |
+|---|---:|---:|---:|
+| SmolLM2-135M | 135M | 2T | 27.13 |
+| **cagliostro-v3** | **146M** | **72.7B** | **26.27** |
+| SmolLM-135M | 135M | 600B | 25.74 |
+| GPT-X2.5-135M | 135M | n/a | 25.17 |
+| Haidass1.5-143M | 143M | n/a | 25.07 |
+| BananaMind-2-Pro | 139M | 100B | 24.96 |
+
+![Index during training](index_curve.png)
+
+The flat stretch from 30B to 64B is not a stall. A warmup-stable-decay schedule holds the learning rate at its peak for the first 85% of the run, and constant peak learning rate depresses zero-shot multiple-choice accuracy even while validation loss keeps improving. The recovery from 22.0 to 26.3 is what the cooldown unlocks, not new knowledge appearing in the last 9B tokens.
+
+![Training loss](loss_curve.png)
+
+The sharp drop in training loss at 63.75B is the data mixture changing, not the model improving. Validation loss on a fixed held-out set is the honest line, and it moves smoothly.
+
+## Model details
+
+| Field | Value |
+|---|---:|
+| Parameters | 146,352,000 |
+| Non-embedding parameters | 85.7% |
+| Layers | 30 |
+| Hidden size | 640 |
+| Intermediate size | 1,536 |
+| Attention heads | 10 |
+| Key/value heads | 5 |
+| Attention | Grouped query attention with cross-head subspace attenuation |
+| Activation | SwiGLU |
+| Normalization | RMSNorm, eps 1e-6 |
+| Positional encoding | RoPE, theta 100,000 |
+| Context length | 2,048 |
+| Vocabulary | 32,768 BPE |
+| Embeddings | Tied input and output |
+| Logit cap | 15.0 |
+| Weights | float32 safetensors |
+
+The architecture is defined in this repository. `trust_remote_code=True` is required because `CagliostroForCausalLM` is not part of transformers.
+
+## Training data
+
+Two mixtures. The first covers the stable phase, the second takes over when the cooldown begins at 85% of the run.
+
+| Source | Stable phase | Cooldown |
+|---|---:|---:|
+| FineWeb-Edu (deduplicated) | 43.7% | 37.0% |
+| DCLM-Baseline | 28.3% | 5.0% |
+| Cosmopedia v2 | 16.0% | 25.0% |
+| FineMath 3+ | 5.0% | 15.0% |
+| OpenMathInstruct-2 | 3.0% | 13.0% |
+| InfiWebMath 3+ | 2.0% | 0.0% |
+| SmolTalk | 2.0% | 5.0% |
+
+Mathematics rises from 10% to 28% during the cooldown. No source exceeds 0.4 epochs across the full run, so nothing is repeated enough to memorize.
+
+## Training setup
+
+| Field | Value |
+|---|---:|
+| Optimizer | AdamW, weight decay 0.01 |
+| Schedule | Warmup-stable-decay |
+| Warmup | 2,000 steps |
+| Stable phase | steps 2,000 to 648,498 at peak learning rate |
+| Cooldown | 114,441 steps, cosine to zero |
+| Tokens per step | 98,304 |
+| Total steps | 762,939 |
+| Precision | bfloat16 with float32 master weights |
+| Hardware | one RTX 5090 |
+| Throughput | 90,000 to 103,000 tokens per second |
+| Wall clock | about 9 days |
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "bench-labs/cagliostro-v3"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype=torch.float32)
+
+ids = tok("The capital of France is", return_tensors="pt")
+out = model.generate(**ids, max_new_tokens=32, do_sample=False)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
+
+This is a base model with no instruction tuning and no chat template. It completes text.
+
+## Reproducing the evaluation
+
+```
+pip install lm-eval
+python -m lm_eval --model hf \
+  --model_args pretrained=bench-labs/cagliostro-v3,dtype=float32,trust_remote_code=True \
+  --tasks hellaswag,arc_easy,arc_challenge,piqa \
+  --num_fewshot 0 --batch_size 8 --device cuda:0
+```
+
+ArithMark-3 uses the script linked from the leaderboard, pointed at the same model id. Evaluate in float32. A bfloat16 round trip moves logits by about 1.3e-1 at this logit cap, which is enough to change borderline multiple-choice answers, while float32 agrees with the training weights to 3.2e-05.
+
+## Provenance
+
+This repository holds the full training history. A checkpoint was pushed every 30 minutes from the first step, giving 363 commits between 11 and 20 September 2026. Any intermediate checkpoint can be retrieved by revision, and the benchmark trajectory in the chart above can be reproduced from them.
+
+## Limitations
+
+English only. 2,048 token context. No instruction tuning, no safety tuning, no RLHF. At 146M parameters it confabulates freely and should not be relied on for factual questions. The mathematics ability measured by ArithMark is arithmetic and short symbolic work, not general mathematical reasoning.
+
+## License
+
+Apache-2.0. The training data is drawn from FineWeb-Edu and FineMath (ODC-By), DCLM-Baseline and OpenMathInstruct-2 (CC-BY-4.0), Cosmopedia v2 and SmolTalk (Apache-2.0).

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "CagliostroForCausalLM"
+  ],
+  "bos_token_id": 0,
+  "canon": false,
+  "dtype": "float32",
+  "eos_token_id": 0,
+  "hidden_size": 640,
+  "intermediate_size": 1536,
+  "layer_repeat": 1,
+  "logit_cap": 15.0,
+  "max_position_embeddings": 2048,
+  "mlp_type": "swiglu",
+  "model_type": "cagliostro",
+  "num_attention_heads": 10,
+  "num_hidden_layers": 30,
+  "num_key_value_heads": 5,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 100000.0,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.17.0",
+  "value_residual": false,
+  "vocab_size": 32768,
+  "xsa": true,
+  "auto_map": {
+    "AutoConfig": "configuration_cagliostro.CagliostroConfig",
+    "AutoModelForCausalLM": "modeling_cagliostro.CagliostroForCausalLM"
+  }
+}

configuration_cagliostro.py

@@ -0,0 +1,43 @@
+from transformers import PretrainedConfig
+
+
+class CagliostroConfig(PretrainedConfig):
+    model_type = "cagliostro"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        vocab_size=32768,
+        hidden_size=640,
+        num_hidden_layers=30,
+        num_attention_heads=10,
+        num_key_value_heads=5,
+        intermediate_size=1600,
+        max_position_embeddings=4096,
+        rope_theta=100000.0,
+        rms_norm_eps=1e-6,
+        tie_word_embeddings=True,
+        mlp_type="swiglu",
+        xsa=False,
+        canon=False,
+        value_residual=False,
+        layer_repeat=1,
+        logit_cap=0.0,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_key_value_heads = num_key_value_heads
+        self.intermediate_size = intermediate_size
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.rms_norm_eps = rms_norm_eps
+        self.mlp_type = mlp_type
+        self.xsa = xsa
+        self.canon = canon
+        self.value_residual = value_residual
+        self.layer_repeat = layer_repeat
+        self.logit_cap = logit_cap
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.17.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11e31616df91cb5b0a4a9a1c352f769f25cb4a2bf6edb4ff221c6295bb5e82a1
+size 585441264

modeling_cagliostro.py

@@ -0,0 +1,270 @@
+import math
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+try:
+    from .configuration_cagliostro import CagliostroConfig
+except ImportError:
+    from configuration_cagliostro import CagliostroConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        dtype = x.dtype
+        x = x.float()
+        x = x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
+        return x.to(dtype) * self.weight
+
+
+def rope_freqs(seq_len, head_dim, theta, device=None):
+    inv_freq = 1.0 / (theta ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
+    t = torch.arange(seq_len, device=device).float()
+    freqs = torch.outer(t, inv_freq)
+    return torch.cat([freqs, freqs], dim=-1)
+
+
+def rotate_half(x):
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat([-x2, x1], dim=-1)
+
+
+def apply_rope(x, cos, sin):
+    return x * cos + rotate_half(x) * sin
+
+
+class Attention(nn.Module):
+    def __init__(self, dim, n_heads, n_kv_heads, head_dim, norm_eps=1e-6,
+                 xsa=False, value_residual=False):
+        super().__init__()
+        if n_heads % n_kv_heads != 0:
+            raise ValueError(
+                f"num_attention_heads ({n_heads}) must be divisible by "
+                f"num_key_value_heads ({n_kv_heads})")
+        self.n_heads = n_heads
+        self.n_kv_heads = n_kv_heads
+        self.head_dim = head_dim
+        self.n_rep = n_heads // n_kv_heads
+        self.xsa = xsa
+
+        self.q_proj = nn.Linear(dim, n_heads * head_dim, bias=False)
+        self.k_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.v_proj = nn.Linear(dim, n_kv_heads * head_dim, bias=False)
+        self.o_proj = nn.Linear(n_heads * head_dim, dim, bias=False)
+
+        self.q_norm = RMSNorm(head_dim, eps=norm_eps)
+        self.k_norm = RMSNorm(head_dim, eps=norm_eps)
+
+        self.value_residual = value_residual
+        if value_residual:
+            self.v_mix = nn.Parameter(torch.full((1,), -4.0))
+
+    def forward(self, x, cos, sin, attn_mask=None, v_first=None):
+        B, T, _ = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim)
+
+        q = self.q_norm(q)
+        k = self.k_norm(k)
+        q = apply_rope(q, cos, sin)
+        k = apply_rope(k, cos, sin)
+
+        v_raw = v
+        if self.value_residual and v_first is not None:
+            lam = torch.sigmoid(self.v_mix)
+            v = (1 - lam) * v + lam * v_first
+
+        q = q.transpose(1, 2)
+        k = k.transpose(1, 2).repeat_interleave(self.n_rep, dim=1)
+        v = v.transpose(1, 2).repeat_interleave(self.n_rep, dim=1)
+
+        if attn_mask is None:
+            out = F.scaled_dot_product_attention(q, k, v, is_causal=True)
+        else:
+            out = F.scaled_dot_product_attention(q, k, v, attn_mask=attn_mask)
+
+        if self.xsa:
+            num = (out * v).sum(dim=-1, keepdim=True)
+            den = (v * v).sum(dim=-1, keepdim=True).clamp_min(1e-6)
+            out = out - (num / den) * v
+
+        out = out.transpose(1, 2).contiguous().view(B, T, -1)
+        return self.o_proj(out), v_raw
+
+
+class SwiGLU(nn.Module):
+    def __init__(self, dim, hidden):
+        super().__init__()
+        self.gate_proj = nn.Linear(dim, hidden, bias=False)
+        self.up_proj = nn.Linear(dim, hidden, bias=False)
+        self.down_proj = nn.Linear(hidden, dim, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class ReLU2MLP(nn.Module):
+    def __init__(self, dim, hidden):
+        super().__init__()
+        self.up_proj = nn.Linear(dim, hidden, bias=False)
+        self.down_proj = nn.Linear(hidden, dim, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.relu(self.up_proj(x)).square())
+
+
+class Canon(nn.Module):
+    def __init__(self, dim, kernel=4):
+        super().__init__()
+        self.kernel = kernel
+        self.conv = nn.Conv1d(dim, dim, kernel, groups=dim, bias=False)
+
+    def forward(self, x):
+        y = F.pad(x.transpose(1, 2), (self.kernel - 1, 0))
+        return x + self.conv(y).transpose(1, 2)
+
+
+class DecoderLayer(nn.Module):
+    def __init__(self, dim, n_heads, n_kv_heads, head_dim, mlp_hidden, norm_eps,
+                 xsa, mlp_type, canon, value_residual):
+        super().__init__()
+        self.attn_norm = RMSNorm(dim, eps=norm_eps)
+        self.attn = Attention(dim, n_heads, n_kv_heads, head_dim, norm_eps,
+                              xsa=xsa, value_residual=value_residual)
+        self.mlp_norm = RMSNorm(dim, eps=norm_eps)
+        self.mlp = ReLU2MLP(dim, mlp_hidden) if mlp_type == "relu2" else SwiGLU(dim, mlp_hidden)
+        self.canon = Canon(dim) if canon else None
+
+    def forward(self, x, cos, sin, attn_mask=None, v_first=None):
+        h = self.attn_norm(x)
+        if self.canon is not None:
+            h = self.canon(h)
+        a, v = self.attn(h, cos, sin, attn_mask, v_first)
+        x = x + a
+        x = x + self.mlp(self.mlp_norm(x))
+        return x, v
+
+
+class CagliostroPreTrainedModel(PreTrainedModel):
+    config_class = CagliostroConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = False
+    _no_split_modules = ["DecoderLayer"]
+    _tied_weights_keys = {"lm_head.weight": "embed_tokens.weight"}
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class CagliostroForCausalLM(CagliostroPreTrainedModel, GenerationMixin):
+    def __init__(self, config):
+        super().__init__(config)
+        dim = config.hidden_size
+        self.head_dim = dim // config.num_attention_heads
+        self.layer_repeat = max(1, config.layer_repeat)
+        self.logit_cap = config.logit_cap
+
+        mlp_hidden = config.intermediate_size
+        if config.mlp_type == "relu2":
+            mlp_hidden = int(round(mlp_hidden * 1.5 / 64) * 64)
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, dim)
+        self.layers = nn.ModuleList([
+            DecoderLayer(
+                dim, config.num_attention_heads, config.num_key_value_heads,
+                self.head_dim, mlp_hidden, config.rms_norm_eps,
+                config.xsa, config.mlp_type, config.canon,
+                config.value_residual and i > 0,
+            )
+            for i in range(config.num_hidden_layers)
+        ])
+        self.norm_out = RMSNorm(dim, eps=config.rms_norm_eps)
+        self.lm_head = nn.Linear(dim, config.vocab_size, bias=False)
+
+        self.rope_theta = config.rope_theta
+        self._rope_cache = None
+
+        self.post_init()
+
+    def _rope(self, seq_len, device, dtype):
+        key = (seq_len, str(device), str(dtype))
+        if self._rope_cache is None or self._rope_cache[0] != key:
+            freqs = rope_freqs(seq_len, self.head_dim, self.rope_theta, device=device)
+            self._rope_cache = (key,
+                                freqs.cos()[None, :, None, :].to(dtype),
+                                freqs.sin()[None, :, None, :].to(dtype))
+        return self._rope_cache[1], self._rope_cache[2]
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, value):
+        self.lm_head = value
+
+    def _causal_mask(self, attention_mask, B, T, dtype, device):
+        causal = torch.ones(T, T, dtype=torch.bool, device=device).tril()
+        mask = causal[None, None, :, :].expand(B, 1, T, T)
+        if attention_mask is not None:
+            pad = attention_mask.to(torch.bool)[:, None, None, :]
+            mask = mask & pad
+        out = torch.zeros(mask.shape, dtype=dtype, device=device)
+        return out.masked_fill(~mask, torch.finfo(dtype).min)
+
+    def forward(self, input_ids=None, attention_mask=None, labels=None,
+                inputs_embeds=None, use_cache=None, output_attentions=None,
+                output_hidden_states=None, return_dict=None, **kwargs):
+        if inputs_embeds is None:
+            x = self.embed_tokens(input_ids)
+        else:
+            x = inputs_embeds
+        B, T, _ = x.shape
+
+        cos, sin = self._rope(T, x.device, x.dtype)
+
+        attn_mask = None
+        if attention_mask is not None and attention_mask.dim() == 2 and not bool(attention_mask.all()):
+            attn_mask = self._causal_mask(attention_mask, B, T, x.dtype, x.device)
+
+        v_first = None
+        for layer in self.layers:
+            for _ in range(self.layer_repeat):
+                x, v = layer(x, cos, sin, attn_mask, v_first)
+                if v_first is None:
+                    v_first = v
+
+        x = self.norm_out(x)
+        logits = self.lm_head(x)
+        if self.logit_cap:
+            logits = self.logit_cap * torch.tanh(logits / self.logit_cap)
+
+        loss = None
+        if labels is not None:
+            shift_logits = logits[:, :-1].float().reshape(-1, logits.size(-1))
+            shift_labels = labels[:, 1:].reshape(-1)
+            loss = F.cross_entropy(shift_logits, shift_labels, ignore_index=-100)
+
+        if return_dict is False:
+            return (loss, logits) if loss is not None else (logits,)
+        return CausalLMOutputWithPast(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **kwargs):
+        return {"input_ids": input_ids, "attention_mask": attention_mask}

tokenizer_config.json

@@ -0,0 +1,7 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>",
+  "model_max_length": 2048
+}