SFT diamond_v3 job 7098234 (checkpoint-396, 3 epochs)

README.md

@@ -1,270 +1,209 @@
 ---
-base_model: openai/gpt-oss-20b
-language:
-- en
-license: apache-2.0
-library_name: transformers
-model_name: ChatTLA-20b
+base_model: EricSpencer00/chattla-20b
+library_name: peft
+pipeline_tag: text-generation
 tags:
-- tla-plus
-- formal-methods
-- formal-verification
-- code-generation
-- trl
+- base_model:adapter:EricSpencer00/chattla-20b
+- lora
 - sft
-- grpo
-- reinforcement-learning
-- generated_from_trainer
-datasets:
-- EricSpencer00/chattla-20b
-pipeline_tag: text-generation
+- transformers
+- trl
 ---
 
-# ChatTLA-20b (v15)
+# Model Card for Model ID
 
-ChatTLA is a fine-tuned version of [openai/gpt-oss-20b](https://huggingface.co/openai/gpt-oss-20b) specialised in generating **TLA+ formal specifications** — the language used by AWS, Microsoft, and Intel to mathematically verify distributed systems.
+<!-- Provide a quick summary of what the model is/does. -->
 
-Given a plain-English description of a concurrent or distributed system, ChatTLA outputs a complete, syntactically valid TLA+ module including `Init`, `Next`, `Spec`, `TypeOK`, and domain invariants, together with a TLC model-checker configuration block.
 
----
 
-## Benchmark Results (v15, 3-shot self-correct)
-
-Evaluated on a 30-spec held-out suite spanning communication protocols, concurrency primitives, consensus, data structures, memory/caches, mutual exclusion, classical puzzles, scheduling, transactions, and workflow state machines. Each spec gets up to 3 self-correction attempts using TLC error feedback. Tiers are defined by what the spec actually does under SANY and TLC, not just whether it parses:
-
-| Tier | Meaning |
-|------|---------|
-| 💎 Diamond | Gold **and** TLC explores ≥1 distinct state, has a non-trivial invariant, and the invariant catches a mutation |
-| 🥇 Gold | SANY parses **and** TLC model-checks clean |
-| 🥈 Silver | SANY parses, TLC finds violation or timeout |
-| Bronze | SANY parse failure |
-
-Diamond is the headline metric: it's the only tier that proves the spec is *semantically* useful rather than just syntactically valid.
-
-### Per-spec results (30-spec holdout)
-
-| # | Batch | Module | Tier | Diamond |
-|---|-------|--------|------|:------:|
-|  1 | communication_protocols | AlternatingBit            | Bronze      |    |
-|  2 | communication_protocols | Arp                       | Bronze      |    |
-|  3 | communication_protocols | AtomicRegister            | Bronze      |    |
-|  4 | concurrency_primitives  | BinarySemaphore           | Bronze      |    |
-|  5 | concurrency_primitives  | Channel                   | Bronze      |    |
-|  6 | concurrency_primitives  | CountDownLatch            | Bronze      |    |
-|  7 | consensus_election      | AtomicCommit              | Bronze      |    |
-|  8 | consensus_election      | BullyElection             | 🥇 Gold     | 💎 |
-|  9 | consensus_election      | ByzantineQuorum           | Bronze      |    |
-| 10 | data_structures         | BinaryHeap                | Bronze      |    |
-| 11 | data_structures         | BloomCounter              | 🥇 Gold     | 💎 |
-| 12 | data_structures         | BloomFilter               | ⏱ Timeout   |    |
-| 13 | memory_caches           | ArenaAllocator            | 🥇 Gold     | 💎 |
-| 14 | memory_caches           | BuddyAllocator            | Bronze      |    |
-| 15 | memory_caches           | CopyingGc                 | Bronze      |    |
-| 16 | mutual_exclusion        | AdaptiveMutex             | 🥇 Gold     | 💎 |
-| 17 | mutual_exclusion        | AndersonMutex             | 🥇 Gold     | 💎 |
-| 18 | mutual_exclusion        | AravindMutex              | ⏱ Timeout   |    |
-| 19 | puzzles_classical       | BlocksWorld               | Bronze      |    |
-| 20 | puzzles_classical       | ChessKingMoves            | Bronze      |    |
-| 21 | puzzles_classical       | ColoredHats               | Bronze      |    |
-| 22 | scheduling_resources    | AdmissionControl          | 🥇 Gold     | 💎 |
-| 23 | scheduling_resources    | BackpressureChannel       | 🥇 Gold     | 💎 |
-| 24 | scheduling_resources    | Bankers                   | ⏱ Timeout   |    |
-| 25 | transactions_databases  | ChainReplication          | ⏱ Timeout   |    |
-| 26 | transactions_databases  | DistributedLock           | Bronze      |    |
-| 27 | transactions_databases  | FencingToken              | Bronze      |    |
-| 28 | workflows_state_machines| ContentModeration         | 🥇 Gold     | 💎 |
-| 29 | workflows_state_machines| DocumentApproval          | 🥇 Gold     | 💎 |
-| 30 | workflows_state_machines| EmailVerification         | Bronze      |    |
-
-**Diamond: 9/30 (30%) · Gold: 9/30 (30%)**
-
-### Per-domain breakdown
-
-| Domain | Diamond |
-|--------|:-------:|
-| communication_protocols | 0/3 |
-| concurrency_primitives | 0/3 |
-| consensus_election | 1/3 |
-| data_structures | 1/3 |
-| memory_caches | 1/3 |
-| mutual_exclusion | 2/3 |
-| puzzles_classical | 0/3 |
-| scheduling_resources | 2/3 |
-| transactions_databases | 0/3 |
-| workflows_state_machines | 2/3 |
-
-### Version history
-
-| Version | Suite | SANY | TLC | Diamond / Notes |
-|---------|-------|------|-----|-----------------|
-| v6  | 20-problem handcraft     | 4/20 (20%)  | 1/20 (5%)  | — |
-| v7  | 20-problem handcraft     | 6/20 (30%)  | 1/20 (5%)  | — |
-| v8  | 20-problem handcraft     | 8/20 (40%)  | 1/20 (5%)  | — |
-| v9  | 20-problem handcraft     | 6/20 (30%)  | 3/20 (15%) | — |
-| v9 best-of-5 + self-correct | 20-problem handcraft | 16/20 (80%) | 5/20 (25%) | — |
-| v10 | 20-problem handcraft     | 6/20 (30%)  | 2/20 (10%) | — |
-| v11 | 20-problem handcraft     | 6/20 (30%)  | 2/20 (10%) | — |
-| v13 (SFT + DPO) | 20-problem handcraft | 9/20 (45%) | 5/20 (25%) | not measured (trivial invariants counted as Gold) |
-| v14 (Diamond SFT) | 30-spec holdout (single-shot) | 16/30 (53%) | 5/30 (17%) | 4/30 (13%) |
-| **v15 (Repair GRPO)** | **30-spec holdout (3-shot)** | 9/30 (30%) | 9/30 (30%) | **9/30 (30%)** |
-
-> v15 applies repair-based GRPO (Group Relative Policy Optimization) on top of v14's Diamond SFT weights. The model learns to fix its own broken specs by training on (broken → repaired) trajectory pairs with TLC-graded improvement reward. v15 eval uses 3-shot self-correction with TLC error feedback, matching realistic usage; v14 was evaluated single-shot, so SANY/TLC rates are not directly comparable. Diamond is the metric to track going forward.
+## Model Details
 
----
+### Model Description
 
-## Quick Start
+<!-- Provide a longer summary of what this model is. -->
 
-### Ollama (recommended)
 
-```bash
-# Pull and run directly
-ollama run EricSpencer00/chattla-20b
 
-# Or use the bundled Modelfile
-curl -L https://huggingface.co/EricSpencer00/chattla-20b/resolve/main/gguf/Modelfile -o Modelfile
-ollama create chattla:20b -f Modelfile
-ollama run chattla:20b "Write a TLA+ spec for a token ring with N nodes."
-```
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-### Python (transformers)
+### Model Sources [optional]
 
-```python
-from transformers import pipeline
+<!-- Provide the basic links for the model. -->
 
-pipe = pipeline(
-    "text-generation",
-    model="EricSpencer00/chattla-20b",
-    device_map="auto",
-)
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-prompt = (
-    "Write a complete TLA+ specification for a two-phase commit protocol "
-    "with one coordinator and N participants."
-)
-result = pipe([{"role": "user", "content": prompt}], max_new_tokens=1024, return_full_text=False)
-print(result[0]["generated_text"])
-```
+## Uses
 
-### llama.cpp / GGUF
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-```bash
-# Download GGUF
-huggingface-cli download EricSpencer00/chattla-20b \
-    gguf/chattla-20b-v15-Q8_0.gguf \
-    --local-dir ./chattla
+### Direct Use
 
-# Run with llama.cpp
-./llama-cli -m chattla/gguf/chattla-20b-v15-Q8_0.gguf \
-    -n 1024 --temp 0.4 \
-    -p "Write a TLA+ spec for mutual exclusion with N processes."
-```
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
----
+[More Information Needed]
 
-## Model Details
+### Downstream Use [optional]
 
-| Property | Value |
-|----------|-------|
-| Base model | openai/gpt-oss-20b |
-| Parameters | 20.9B |
-| Architecture | GptOss (sliding + full attention) |
-| Fine-tuning method | Diamond SFT (LoRA) → Repair GRPO (LoRA) → merged |
-| Context length | 2048 (trained) / 131072 (base) |
-| GGUF quantisation | Q8_0 (~22 GB) |
-| Training date | April 2026 |
-
-### System prompt
-
-The model is prompted with:
-
-```
-You are ChatTLA, an expert at writing verified TLA+ formal specifications.
-When asked to write a TLA+ spec, follow these rules exactly:
-1. Start the module with ---- MODULE <ModuleName> ----
-2. End with ====
-3. Include EXTENDS, VARIABLES, Init, Next, and Spec operators
-4. After the TLA+ module, append a TLC configuration block:
-   SPECIFICATION Spec
-   INVARIANT TypeOK   (if TypeOK is defined)
-5. Output only valid TLA+ code. No markdown fences, no explanation outside the spec.
-```
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
----
+[More Information Needed]
 
-## Training
+### Out-of-Scope Use
 
-### Phase 1: Diamond SFT (v14)
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-v14 was produced by the **Diamond curation pipeline**: candidate TLA+ specs are generated by an earlier checkpoint, then graded by a tlc_validator that checks SANY parsing, TLC state-space exploration, non-trivial invariants, and mutation-test sensitivity. Specs that survive grading are LLM-judged for chain-of-thought quality, leaving a curated training pool (209 raw → 73 curated for the v14 SFT round). The model is fine-tuned with LoRA on this pool and merged.
+[More Information Needed]
 
-### Phase 2: Repair GRPO (v15)
+## Bias, Risks, and Limitations
 
-v15 applies **repair-based GRPO** (Group Relative Policy Optimization) on top of the v14 checkpoint. The key insight: instead of training on gold-standard specs alone, the model learns to *fix broken specs* using TLC error feedback as reward signal.
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-**Pipeline:**
-1. **Trajectory collection** — the v14 model generates specs for 398 problems with up to 6 repair iterations each, producing (broken, repaired) pairs scored by a multi-stage validator (SANY → TLC → Apalache → TLAPS).
-2. **Dataset filtering** — pairs are filtered to keep the "learnable middle": `min_before_score=0.10` (drop unparseable) and `max_before_score=0.80` (drop already-good), yielding ~430 gradable pairs centered on score ≈ 0.45.
-3. **GRPO training** — 300 steps, 4 generations per prompt, max 384 completion tokens. The reward is the improvement delta: `after_score - before_score`, normalized by group. Learning rate 3e-6, KL penalty β=0.02, temperature 0.5.
-4. **LoRA merge** — best checkpoint (around step 140–160 where reward peaked) merged back into full weights.
+[More Information Needed]
 
-Reward peaked at steps 140–160 with `reward_std ≈ 0.25` (vs 0.0 in prior full-spec GRPO attempts that had zero variance). This was the first successful RL run on TLA+ spec generation.
+### Recommendations
 
-**R2 regression and R3 (in progress).** A second flywheel round (R2) continued GRPO from v15's merged weights on a freshly harvested dataset and regressed to 6/30 (20%). Post-mortem: the Phase 2 merge deduped pairs on `(nl[:80], round(before_score, 1))`, a score-bucket width of 0.1 that collapsed most of the learnable-middle band; combined with a raised `min_before_score = 0.10`, the usable training set fell from 433 → 179 pairs, shifted hard (mean before_score 0.26 → 0.42), and the model overtrained past its 150-step peak over 300 steps. Regressions concentrated in `mutual_exclusion` and `workflows_state_machines` (2/3 lost each). R3 pulls only the data and step-budget levers: dedup key widened to `(nl[:120], round(before_score, 2))`, score floor restored to 0.02, `--max-iters` raised 6 → 9 to grow the raw pool, and `--max-steps` cut to 175 with a checkpoint picker that selects the save closest to step 150. v15 remains the production checkpoint until R3 beats 9/30.
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-DPO/KTO refinement was used in v11–v13 but was deprecated in the Diamond overhaul: 0/484 specs from those preference-trained checkpoints actually passed Diamond, indicating the model had learned TLA+ syntax without learning semantics.
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-### Training configuration
+## How to Get Started with the Model
 
-| Setting | Value |
-|---------|-------|
-| SFT method | LoRA (lora_dropout=0) |
-| GRPO method | LoRA, 4 generations, 384 max completion |
-| GRPO learning rate | 3e-6 |
-| GRPO KL β | 0.02 |
-| GRPO steps | 300 (best checkpoint ~150) |
-| Max sequence length | 2048 |
-| TRL | 0.28.0 |
-| Transformers | 5.2.0 |
-| PyTorch | 2.10.0 |
-| Hardware | 2× Quadro RTX 8000 (48 GB each) |
+Use the code below to get started with the model.
 
----
+[More Information Needed]
 
-## Files
-
-```
-EricSpencer00/chattla-20b
-├── config.json              # Model architecture
-├── tokenizer.json           # Tokenizer
-├── tokenizer_config.json
-├── chat_template.jinja      # Chat template
-├── pytorch_model.bin        # Full BF16 weights (39 GB)
-├── generation_config.json
-└── gguf/
-    ├── chattla-20b-v15-Q8_0.gguf   # Quantised GGUF for Ollama / llama.cpp
-    └── Modelfile                    # Ollama Modelfile
-```
+## Training Details
 
----
+### Training Data
 
-## Intended Use
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-ChatTLA is designed for:
-- Rapid prototyping of TLA+ specifications from natural-language system descriptions
-- Educational exploration of formal methods
-- Assisting engineers who are learning TLA+
+[More Information Needed]
 
-**Not intended for:** safety-critical or production verification without human review. Always validate generated specs with SANY and TLC before relying on them.
+### Training Procedure
 
----
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
 
-## Citation
+[More Information Needed]
+### Framework versions
 
-```bibtex
-@misc{chattla2026,
-  title   = {ChatTLA: Fine-tuned LLM for TLA+ Formal Specification Generation},
-  author  = {Spencer, Eric},
-  year    = {2026},
-  url     = {https://huggingface.co/EricSpencer00/chattla-20b},
-}
-```
+- PEFT 0.19.1

adapter_config.json

@@ -0,0 +1,110 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "EricSpencer00/chattla-20b",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": [
+    0,
+    1,
+    2,
+    3,
+    4,
+    5,
+    6,
+    7,
+    8,
+    9,
+    10,
+    11,
+    12,
+    13,
+    14,
+    15,
+    16,
+    17,
+    18,
+    19,
+    20,
+    21,
+    22,
+    23,
+    24,
+    25,
+    26,
+    27,
+    28,
+    29,
+    30,
+    31,
+    32,
+    33,
+    34,
+    35,
+    36,
+    37,
+    38,
+    39,
+    40,
+    41,
+    42,
+    43,
+    44,
+    45,
+    46,
+    47,
+    48,
+    49,
+    50,
+    51,
+    52,
+    53,
+    54,
+    55,
+    56,
+    57,
+    58,
+    59,
+    60,
+    61,
+    62,
+    63
+  ],
+  "loftq_config": {},
+  "lora_alpha": 16,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.19.1",
+  "qalora_group_size": 16,
+  "r": 8,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "v_proj",
+    "q_proj",
+    "k_proj",
+    "o_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f6af38d6fd9761f7616c0e4ff11847fb3b879987b66e5b441c55431173886fc5
+size 7988016

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50898c016cfce6af2ef195d037fee70e17813fdbe2e83b38e9d5f9889f697684
+size 16089611

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a9f217e852f439efa6bd32fde98d6867f11aa6ea13ddc021ba10af6a0b0934
+size 14645

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9e67916fd5f503c55c9f912b2f0e02a7346b0c165f620bf4d5255975c891e22
+size 1465

tokenizer_config.json

@@ -3,7 +3,8 @@
   "bos_token": "<|startoftext|>",
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|return|>",
-  "is_local": true,
+  "is_local": false,
+  "local_files_only": false,
   "model_input_names": [
     "input_ids",
     "attention_mask"

trainer_state.json

@@ -0,0 +1,824 @@
+{
+  "best_global_step": null,
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 3.0,
+  "eval_steps": 500,
+  "global_step": 396,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "entropy": 1.8488737344741821,
+      "epoch": 0.03787878787878788,
+      "grad_norm": 2.390625,
+      "learning_rate": 8e-05,
+      "loss": 2.6958173751831054,
+      "mean_token_accuracy": 0.5450957447290421,
+      "num_tokens": 35961.0,
+      "step": 5
+    },
+    {
+      "entropy": 1.7560989141464234,
+      "epoch": 0.07575757575757576,
+      "grad_norm": 4.90625,
+      "learning_rate": 9.997417925081962e-05,
+      "loss": 2.22491512298584,
+      "mean_token_accuracy": 0.6006776213645935,
+      "num_tokens": 73396.0,
+      "step": 10
+    },
+    {
+      "entropy": 1.4386030733585358,
+      "epoch": 0.11363636363636363,
+      "grad_norm": 2.578125,
+      "learning_rate": 9.986932816177258e-05,
+      "loss": 1.6638334274291993,
+      "mean_token_accuracy": 0.672905695438385,
+      "num_tokens": 109782.0,
+      "step": 15
+    },
+    {
+      "entropy": 1.3276323318481444,
+      "epoch": 0.15151515151515152,
+      "grad_norm": 1.1796875,
+      "learning_rate": 9.96840020059622e-05,
+      "loss": 1.4536287307739257,
+      "mean_token_accuracy": 0.6756029307842255,
+      "num_tokens": 143815.0,
+      "step": 20
+    },
+    {
+      "entropy": 1.389198613166809,
+      "epoch": 0.1893939393939394,
+      "grad_norm": 0.88671875,
+      "learning_rate": 9.94184998476693e-05,
+      "loss": 1.4698452949523926,
+      "mean_token_accuracy": 0.6868252992630005,
+      "num_tokens": 183913.0,
+      "step": 25
+    },
+    {
+      "entropy": 1.2845717668533325,
+      "epoch": 0.22727272727272727,
+      "grad_norm": 0.98046875,
+      "learning_rate": 9.907325013268815e-05,
+      "loss": 1.267971897125244,
+      "mean_token_accuracy": 0.7166933000087738,
+      "num_tokens": 220563.0,
+      "step": 30
+    },
+    {
+      "entropy": 1.219918441772461,
+      "epoch": 0.26515151515151514,
+      "grad_norm": 1.0078125,
+      "learning_rate": 9.864880999693551e-05,
+      "loss": 1.2410313606262207,
+      "mean_token_accuracy": 0.7290079295635223,
+      "num_tokens": 257973.0,
+      "step": 35
+    },
+    {
+      "entropy": 0.9539740681648254,
+      "epoch": 0.30303030303030304,
+      "grad_norm": 1.453125,
+      "learning_rate": 9.814586436738998e-05,
+      "loss": 0.9286725997924805,
+      "mean_token_accuracy": 0.7912321865558625,
+      "num_tokens": 289221.0,
+      "step": 40
+    },
+    {
+      "entropy": 0.8536828875541687,
+      "epoch": 0.3409090909090909,
+      "grad_norm": 1.0546875,
+      "learning_rate": 9.756522485681247e-05,
+      "loss": 0.9232993125915527,
+      "mean_token_accuracy": 0.7938548445701599,
+      "num_tokens": 328397.0,
+      "step": 45
+    },
+    {
+      "entropy": 0.7762708187103271,
+      "epoch": 0.3787878787878788,
+      "grad_norm": 1.03125,
+      "learning_rate": 9.690782845403164e-05,
+      "loss": 0.7707252502441406,
+      "mean_token_accuracy": 0.8317423284053802,
+      "num_tokens": 363591.0,
+      "step": 50
+    },
+    {
+      "entropy": 0.7853556156158448,
+      "epoch": 0.4166666666666667,
+      "grad_norm": 1.3671875,
+      "learning_rate": 9.617473601190743e-05,
+      "loss": 0.7815133094787597,
+      "mean_token_accuracy": 0.8228329360485077,
+      "num_tokens": 396629.0,
+      "step": 55
+    },
+    {
+      "entropy": 0.8222164928913116,
+      "epoch": 0.45454545454545453,
+      "grad_norm": 0.77734375,
+      "learning_rate": 9.5367130535413e-05,
+      "loss": 0.826558780670166,
+      "mean_token_accuracy": 0.8203958511352539,
+      "num_tokens": 436793.0,
+      "step": 60
+    },
+    {
+      "entropy": 0.7317156493663788,
+      "epoch": 0.49242424242424243,
+      "grad_norm": 1.8515625,
+      "learning_rate": 9.448631527259749e-05,
+      "loss": 0.7245882034301758,
+      "mean_token_accuracy": 0.8379499971866607,
+      "num_tokens": 471517.0,
+      "step": 65
+    },
+    {
+      "entropy": 0.6935560047626496,
+      "epoch": 0.5303030303030303,
+      "grad_norm": 1.0,
+      "learning_rate": 9.353371161151032e-05,
+      "loss": 0.737287950515747,
+      "mean_token_accuracy": 0.8402327358722687,
+      "num_tokens": 509690.0,
+      "step": 70
+    },
+    {
+      "entropy": 0.7783978283405304,
+      "epoch": 0.5681818181818182,
+      "grad_norm": 0.72265625,
+      "learning_rate": 9.251085678648072e-05,
+      "loss": 0.8002090454101562,
+      "mean_token_accuracy": 0.8231807351112366,
+      "num_tokens": 548057.0,
+      "step": 75
+    },
+    {
+      "entropy": 0.6315798878669738,
+      "epoch": 0.6060606060606061,
+      "grad_norm": 0.76171875,
+      "learning_rate": 9.14194013974539e-05,
+      "loss": 0.612918758392334,
+      "mean_token_accuracy": 0.855874752998352,
+      "num_tokens": 586733.0,
+      "step": 80
+    },
+    {
+      "entropy": 0.6434627115726471,
+      "epoch": 0.6439393939393939,
+      "grad_norm": 0.71875,
+      "learning_rate": 9.026110674638721e-05,
+      "loss": 0.651760482788086,
+      "mean_token_accuracy": 0.8549414694309234,
+      "num_tokens": 620130.0,
+      "step": 85
+    },
+    {
+      "entropy": 0.7037500739097595,
+      "epoch": 0.6818181818181818,
+      "grad_norm": 0.81640625,
+      "learning_rate": 8.903784199500411e-05,
+      "loss": 0.7022861957550048,
+      "mean_token_accuracy": 0.8460219502449036,
+      "num_tokens": 653599.0,
+      "step": 90
+    },
+    {
+      "entropy": 0.6765074074268341,
+      "epoch": 0.7196969696969697,
+      "grad_norm": 0.875,
+      "learning_rate": 8.77515811484931e-05,
+      "loss": 0.6895392894744873,
+      "mean_token_accuracy": 0.841349971294403,
+      "num_tokens": 690828.0,
+      "step": 95
+    },
+    {
+      "entropy": 0.5865823537111282,
+      "epoch": 0.7575757575757576,
+      "grad_norm": 0.6953125,
+      "learning_rate": 8.640439987001853e-05,
+      "loss": 0.5448073863983154,
+      "mean_token_accuracy": 0.8746069610118866,
+      "num_tokens": 727387.0,
+      "step": 100
+    },
+    {
+      "entropy": 0.5738832741975785,
+      "epoch": 0.7954545454545454,
+      "grad_norm": 0.88671875,
+      "learning_rate": 8.49984721311843e-05,
+      "loss": 0.5894301891326904,
+      "mean_token_accuracy": 0.8635632038116455,
+      "num_tokens": 763245.0,
+      "step": 105
+    },
+    {
+      "entropy": 0.6615035921335221,
+      "epoch": 0.8333333333333334,
+      "grad_norm": 0.6953125,
+      "learning_rate": 8.353606670385515e-05,
+      "loss": 0.6997550010681153,
+      "mean_token_accuracy": 0.8429525136947632,
+      "num_tokens": 797170.0,
+      "step": 110
+    },
+    {
+      "entropy": 0.5302515834569931,
+      "epoch": 0.8712121212121212,
+      "grad_norm": 0.62109375,
+      "learning_rate": 8.201954349899713e-05,
+      "loss": 0.5376251697540283,
+      "mean_token_accuracy": 0.8770341098308563,
+      "num_tokens": 829505.0,
+      "step": 115
+    },
+    {
+      "entropy": 0.575181958079338,
+      "epoch": 0.9090909090909091,
+      "grad_norm": 0.70703125,
+      "learning_rate": 8.04513497584452e-05,
+      "loss": 0.5972713947296142,
+      "mean_token_accuracy": 0.8632317066192627,
+      "num_tokens": 865751.0,
+      "step": 120
+    },
+    {
+      "entropy": 0.6076352208852768,
+      "epoch": 0.946969696969697,
+      "grad_norm": 0.60546875,
+      "learning_rate": 7.883401610574336e-05,
+      "loss": 0.602271556854248,
+      "mean_token_accuracy": 0.8631208717823029,
+      "num_tokens": 901386.0,
+      "step": 125
+    },
+    {
+      "entropy": 0.6081637680530548,
+      "epoch": 0.9848484848484849,
+      "grad_norm": 0.828125,
+      "learning_rate": 7.717015246243011e-05,
+      "loss": 0.6359455585479736,
+      "mean_token_accuracy": 0.8557994425296783,
+      "num_tokens": 939393.0,
+      "step": 130
+    },
+    {
+      "entropy": 0.5041210368275643,
+      "epoch": 1.0227272727272727,
+      "grad_norm": 0.91015625,
+      "learning_rate": 7.546244383635928e-05,
+      "loss": 0.5249661445617676,
+      "mean_token_accuracy": 0.8816903054714202,
+      "num_tokens": 973074.0,
+      "step": 135
+    },
+    {
+      "entropy": 0.5709114253520966,
+      "epoch": 1.0606060606060606,
+      "grad_norm": 0.59375,
+      "learning_rate": 7.371364598885276e-05,
+      "loss": 0.5937416553497314,
+      "mean_token_accuracy": 0.8642938137054443,
+      "num_tokens": 1008969.0,
+      "step": 140
+    },
+    {
+      "entropy": 0.6661858707666397,
+      "epoch": 1.0984848484848484,
+      "grad_norm": 0.74609375,
+      "learning_rate": 7.192658098767686e-05,
+      "loss": 0.6871523857116699,
+      "mean_token_accuracy": 0.8403465390205384,
+      "num_tokens": 1048099.0,
+      "step": 145
+    },
+    {
+      "entropy": 0.6694399774074554,
+      "epoch": 1.1363636363636362,
+      "grad_norm": 1.234375,
+      "learning_rate": 7.010413265301888e-05,
+      "loss": 0.6669593811035156,
+      "mean_token_accuracy": 0.8518058300018311,
+      "num_tokens": 1088002.0,
+      "step": 150
+    },
+    {
+      "entropy": 0.5742821432650089,
+      "epoch": 1.1742424242424243,
+      "grad_norm": 0.671875,
+      "learning_rate": 6.824924190381256e-05,
+      "loss": 0.579827356338501,
+      "mean_token_accuracy": 0.8686598777770996,
+      "num_tokens": 1125008.0,
+      "step": 155
+    },
+    {
+      "entropy": 0.5425577774643898,
+      "epoch": 1.2121212121212122,
+      "grad_norm": 0.8125,
+      "learning_rate": 6.63649020119223e-05,
+      "loss": 0.5697774887084961,
+      "mean_token_accuracy": 0.8714725434780121,
+      "num_tokens": 1160396.0,
+      "step": 160
+    },
+    {
+      "entropy": 0.5542260497808457,
+      "epoch": 1.25,
+      "grad_norm": 0.88671875,
+      "learning_rate": 6.445415377184427e-05,
+      "loss": 0.5461042404174805,
+      "mean_token_accuracy": 0.8678983390331269,
+      "num_tokens": 1193022.0,
+      "step": 165
+    },
+    {
+      "entropy": 0.6764177441596985,
+      "epoch": 1.2878787878787878,
+      "grad_norm": 1.0546875,
+      "learning_rate": 6.252008059371968e-05,
+      "loss": 0.6604482173919678,
+      "mean_token_accuracy": 0.8478662848472596,
+      "num_tokens": 1230547.0,
+      "step": 170
+    },
+    {
+      "entropy": 0.5709842652082443,
+      "epoch": 1.3257575757575757,
+      "grad_norm": 0.921875,
+      "learning_rate": 6.056580352757813e-05,
+      "loss": 0.5742652416229248,
+      "mean_token_accuracy": 0.8663301408290863,
+      "num_tokens": 1269514.0,
+      "step": 175
+    },
+    {
+      "entropy": 0.5634667783975601,
+      "epoch": 1.3636363636363638,
+      "grad_norm": 0.8203125,
+      "learning_rate": 5.8594476226840835e-05,
+      "loss": 0.5817259788513184,
+      "mean_token_accuracy": 0.8571344554424286,
+      "num_tokens": 1305450.0,
+      "step": 180
+    },
+    {
+      "entropy": 0.5020789414644241,
+      "epoch": 1.4015151515151514,
+      "grad_norm": 0.890625,
+      "learning_rate": 5.660927985921122e-05,
+      "loss": 0.5025756359100342,
+      "mean_token_accuracy": 0.8809928238391876,
+      "num_tokens": 1342076.0,
+      "step": 185
+    },
+    {
+      "entropy": 0.5636127024888993,
+      "epoch": 1.4393939393939394,
+      "grad_norm": 0.74609375,
+      "learning_rate": 5.4613417973165106e-05,
+      "loss": 0.5667418956756591,
+      "mean_token_accuracy": 0.8666031301021576,
+      "num_tokens": 1383914.0,
+      "step": 190
+    },
+    {
+      "entropy": 0.5760275781154632,
+      "epoch": 1.4772727272727273,
+      "grad_norm": 0.96484375,
+      "learning_rate": 5.26101113283247e-05,
+      "loss": 0.594722318649292,
+      "mean_token_accuracy": 0.8600034713745117,
+      "num_tokens": 1416350.0,
+      "step": 195
+    },
+    {
+      "entropy": 0.5440281018614769,
+      "epoch": 1.5151515151515151,
+      "grad_norm": 0.890625,
+      "learning_rate": 5.06025926980586e-05,
+      "loss": 0.5633892059326172,
+      "mean_token_accuracy": 0.8706209361553192,
+      "num_tokens": 1450795.0,
+      "step": 200
+    },
+    {
+      "entropy": 0.5666765511035919,
+      "epoch": 1.553030303030303,
+      "grad_norm": 0.8203125,
+      "learning_rate": 4.859410165269499e-05,
+      "loss": 0.5622632503509521,
+      "mean_token_accuracy": 0.8659008502960205,
+      "num_tokens": 1486675.0,
+      "step": 205
+    },
+    {
+      "entropy": 0.5974125057458878,
+      "epoch": 1.5909090909090908,
+      "grad_norm": 0.8984375,
+      "learning_rate": 4.658787933176646e-05,
+      "loss": 0.6370823383331299,
+      "mean_token_accuracy": 0.8563625752925873,
+      "num_tokens": 1523640.0,
+      "step": 210
+    },
+    {
+      "entropy": 0.4951909102499485,
+      "epoch": 1.628787878787879,
+      "grad_norm": 2.5625,
+      "learning_rate": 4.458716321372259e-05,
+      "loss": 0.5116774082183838,
+      "mean_token_accuracy": 0.8819014012813569,
+      "num_tokens": 1555473.0,
+      "step": 215
+    },
+    {
+      "entropy": 0.4781244724988937,
+      "epoch": 1.6666666666666665,
+      "grad_norm": 0.796875,
+      "learning_rate": 4.259518189155048e-05,
+      "loss": 0.4745966911315918,
+      "mean_token_accuracy": 0.8877918183803558,
+      "num_tokens": 1588856.0,
+      "step": 220
+    },
+    {
+      "entropy": 0.5352629214525223,
+      "epoch": 1.7045454545454546,
+      "grad_norm": 0.80859375,
+      "learning_rate": 4.0615149862733907e-05,
+      "loss": 0.5249689102172852,
+      "mean_token_accuracy": 0.8741750419139862,
+      "num_tokens": 1623594.0,
+      "step": 225
+    },
+    {
+      "entropy": 0.691879364848137,
+      "epoch": 1.7424242424242424,
+      "grad_norm": 1.046875,
+      "learning_rate": 3.8650262341958627e-05,
+      "loss": 0.7096127510070801,
+      "mean_token_accuracy": 0.8415236473083496,
+      "num_tokens": 1663579.0,
+      "step": 230
+    },
+    {
+      "entropy": 0.5149153739213943,
+      "epoch": 1.7803030303030303,
+      "grad_norm": 0.67578125,
+      "learning_rate": 3.6703690104934804e-05,
+      "loss": 0.5001875400543213,
+      "mean_token_accuracy": 0.879623144865036,
+      "num_tokens": 1699916.0,
+      "step": 235
+    },
+    {
+      "entropy": 0.3942973747849464,
+      "epoch": 1.8181818181818183,
+      "grad_norm": 0.78515625,
+      "learning_rate": 3.477857437165694e-05,
+      "loss": 0.37698049545288087,
+      "mean_token_accuracy": 0.9060261309146881,
+      "num_tokens": 1729598.0,
+      "step": 240
+    },
+    {
+      "entropy": 0.5332215487957,
+      "epoch": 1.856060606060606,
+      "grad_norm": 0.7109375,
+      "learning_rate": 3.2878021737358474e-05,
+      "loss": 0.5564140796661377,
+      "mean_token_accuracy": 0.8692101180553437,
+      "num_tokens": 1765449.0,
+      "step": 245
+    },
+    {
+      "entropy": 0.5327667541801929,
+      "epoch": 1.893939393939394,
+      "grad_norm": 0.79296875,
+      "learning_rate": 3.100509915934104e-05,
+      "loss": 0.551155424118042,
+      "mean_token_accuracy": 0.8772011756896972,
+      "num_tokens": 1799864.0,
+      "step": 250
+    },
+    {
+      "entropy": 0.5360715672373771,
+      "epoch": 1.9318181818181817,
+      "grad_norm": 0.79296875,
+      "learning_rate": 2.91628290077681e-05,
+      "loss": 0.5367880821228027,
+      "mean_token_accuracy": 0.870383208990097,
+      "num_tokens": 1833622.0,
+      "step": 255
+    },
+    {
+      "entropy": 0.5413278043270111,
+      "epoch": 1.9696969696969697,
+      "grad_norm": 0.91796875,
+      "learning_rate": 2.735418418840977e-05,
+      "loss": 0.5825323104858399,
+      "mean_token_accuracy": 0.87061527967453,
+      "num_tokens": 1873310.0,
+      "step": 260
+    },
+    {
+      "entropy": 0.5841432213783264,
+      "epoch": 2.007575757575758,
+      "grad_norm": 0.8203125,
+      "learning_rate": 2.5582083345209217e-05,
+      "loss": 0.5896332740783692,
+      "mean_token_accuracy": 0.8582423806190491,
+      "num_tokens": 1911297.0,
+      "step": 265
+    },
+    {
+      "entropy": 0.4948613867163658,
+      "epoch": 2.0454545454545454,
+      "grad_norm": 0.69140625,
+      "learning_rate": 2.3849386150412378e-05,
+      "loss": 0.5073411464691162,
+      "mean_token_accuracy": 0.8816523849964142,
+      "num_tokens": 1946721.0,
+      "step": 270
+    },
+    {
+      "entropy": 0.5449469789862633,
+      "epoch": 2.0833333333333335,
+      "grad_norm": 0.98828125,
+      "learning_rate": 2.2158888689861433e-05,
+      "loss": 0.55238938331604,
+      "mean_token_accuracy": 0.870269101858139,
+      "num_tokens": 1983131.0,
+      "step": 275
+    },
+    {
+      "entropy": 0.5343923151493073,
+      "epoch": 2.121212121212121,
+      "grad_norm": 0.828125,
+      "learning_rate": 2.051331895089882e-05,
+      "loss": 0.5471882343292236,
+      "mean_token_accuracy": 0.8725048422813415,
+      "num_tokens": 2022170.0,
+      "step": 280
+    },
+    {
+      "entropy": 0.5085220277309418,
+      "epoch": 2.159090909090909,
+      "grad_norm": 0.859375,
+      "learning_rate": 1.8915332420163073e-05,
+      "loss": 0.5496640682220459,
+      "mean_token_accuracy": 0.8838891804218292,
+      "num_tokens": 2056605.0,
+      "step": 285
+    },
+    {
+      "entropy": 0.5516770869493485,
+      "epoch": 2.196969696969697,
+      "grad_norm": 0.76953125,
+      "learning_rate": 1.736750779838044e-05,
+      "loss": 0.556769609451294,
+      "mean_token_accuracy": 0.866687560081482,
+      "num_tokens": 2093824.0,
+      "step": 290
+    },
+    {
+      "entropy": 0.45218280255794524,
+      "epoch": 2.234848484848485,
+      "grad_norm": 0.8046875,
+      "learning_rate": 1.5872342839067306e-05,
+      "loss": 0.45604901313781737,
+      "mean_token_accuracy": 0.8922043800354004,
+      "num_tokens": 2126566.0,
+      "step": 295
+    },
+    {
+      "entropy": 0.3802251495420933,
+      "epoch": 2.2727272727272725,
+      "grad_norm": 0.8671875,
+      "learning_rate": 1.4432250317858675e-05,
+      "loss": 0.38388445377349856,
+      "mean_token_accuracy": 0.912166029214859,
+      "num_tokens": 2157003.0,
+      "step": 300
+    },
+    {
+      "entropy": 0.4966444715857506,
+      "epoch": 2.3106060606060606,
+      "grad_norm": 0.953125,
+      "learning_rate": 1.3049554138967051e-05,
+      "loss": 0.5617212295532227,
+      "mean_token_accuracy": 0.8805790543556213,
+      "num_tokens": 2196031.0,
+      "step": 305
+    },
+    {
+      "entropy": 0.4347878597676754,
+      "epoch": 2.3484848484848486,
+      "grad_norm": 0.78125,
+      "learning_rate": 1.172648558505477e-05,
+      "loss": 0.44159908294677735,
+      "mean_token_accuracy": 0.8970139145851135,
+      "num_tokens": 2230559.0,
+      "step": 310
+    },
+    {
+      "entropy": 0.5470431834459305,
+      "epoch": 2.3863636363636362,
+      "grad_norm": 0.89453125,
+      "learning_rate": 1.0465179716571466e-05,
+      "loss": 0.5724119663238525,
+      "mean_token_accuracy": 0.8691107213497162,
+      "num_tokens": 2268041.0,
+      "step": 315
+    },
+    {
+      "entropy": 0.5263485819101333,
+      "epoch": 2.4242424242424243,
+      "grad_norm": 0.96875,
+      "learning_rate": 9.267671926367166e-06,
+      "loss": 0.5039608955383301,
+      "mean_token_accuracy": 0.8749733746051789,
+      "num_tokens": 2302515.0,
+      "step": 320
+    },
+    {
+      "entropy": 0.6496276021003723,
+      "epoch": 2.462121212121212,
+      "grad_norm": 0.8359375,
+      "learning_rate": 8.135894655140758e-06,
+      "loss": 0.6695892810821533,
+      "mean_token_accuracy": 0.8468307316303253,
+      "num_tokens": 2343731.0,
+      "step": 325
+    },
+    {
+      "entropy": 0.49519053399562835,
+      "epoch": 2.5,
+      "grad_norm": 0.91015625,
+      "learning_rate": 7.071674273024354e-06,
+      "loss": 0.5048277378082275,
+      "mean_token_accuracy": 0.8870163023471832,
+      "num_tokens": 2380175.0,
+      "step": 330
+    },
+    {
+      "entropy": 0.48127582371234895,
+      "epoch": 2.537878787878788,
+      "grad_norm": 1.0078125,
+      "learning_rate": 6.076728132335669e-06,
+      "loss": 0.492659330368042,
+      "mean_token_accuracy": 0.8882244884967804,
+      "num_tokens": 2412514.0,
+      "step": 335
+    },
+    {
+      "entropy": 0.46426295340061186,
+      "epoch": 2.5757575757575757,
+      "grad_norm": 0.74609375,
+      "learning_rate": 5.152661796254504e-06,
+      "loss": 0.46677451133728026,
+      "mean_token_accuracy": 0.8887528300285339,
+      "num_tokens": 2444160.0,
+      "step": 340
+    },
+    {
+      "entropy": 0.5339648842811584,
+      "epoch": 2.6136363636363638,
+      "grad_norm": 1.1796875,
+      "learning_rate": 4.300966447895438e-06,
+      "loss": 0.5490293502807617,
+      "mean_token_accuracy": 0.8735897600650787,
+      "num_tokens": 2481155.0,
+      "step": 345
+    },
+    {
+      "entropy": 0.5200017690658569,
+      "epoch": 2.6515151515151514,
+      "grad_norm": 1.0,
+      "learning_rate": 3.5230164839577416e-06,
+      "loss": 0.5357878684997559,
+      "mean_token_accuracy": 0.8745341539382935,
+      "num_tokens": 2516557.0,
+      "step": 350
+    },
+    {
+      "entropy": 0.40029837638139726,
+      "epoch": 2.6893939393939394,
+      "grad_norm": 0.81640625,
+      "learning_rate": 2.820067296835799e-06,
+      "loss": 0.40105533599853516,
+      "mean_token_accuracy": 0.9027827084064484,
+      "num_tokens": 2546964.0,
+      "step": 355
+    },
+    {
+      "entropy": 0.5041794210672379,
+      "epoch": 2.7272727272727275,
+      "grad_norm": 0.890625,
+      "learning_rate": 2.1932532487688785e-06,
+      "loss": 0.5031059265136719,
+      "mean_token_accuracy": 0.8806142807006836,
+      "num_tokens": 2582064.0,
+      "step": 360
+    },
+    {
+      "entropy": 0.5365766167640686,
+      "epoch": 2.765151515151515,
+      "grad_norm": 1.125,
+      "learning_rate": 1.6435858412996275e-06,
+      "loss": 0.5377714157104492,
+      "mean_token_accuracy": 0.8711750507354736,
+      "num_tokens": 2620085.0,
+      "step": 365
+    },
+    {
+      "entropy": 0.5815416038036346,
+      "epoch": 2.8030303030303028,
+      "grad_norm": 0.80078125,
+      "learning_rate": 1.1719520829951203e-06,
+      "loss": 0.5831816673278809,
+      "mean_token_accuracy": 0.8631684601306915,
+      "num_tokens": 2655969.0,
+      "step": 370
+    },
+    {
+      "entropy": 0.6044020265340805,
+      "epoch": 2.840909090909091,
+      "grad_norm": 0.86328125,
+      "learning_rate": 7.791130580645622e-07,
+      "loss": 0.6402379035949707,
+      "mean_token_accuracy": 0.8542465031147003,
+      "num_tokens": 2698335.0,
+      "step": 375
+    },
+    {
+      "entropy": 0.5233386039733887,
+      "epoch": 2.878787878787879,
+      "grad_norm": 0.8828125,
+      "learning_rate": 4.6570269818346224e-07,
+      "loss": 0.531651782989502,
+      "mean_token_accuracy": 0.8721617937088013,
+      "num_tokens": 2733837.0,
+      "step": 380
+    },
+    {
+      "entropy": 0.6269441485404968,
+      "epoch": 2.9166666666666665,
+      "grad_norm": 0.8046875,
+      "learning_rate": 2.3222675950627104e-07,
+      "loss": 0.6280055522918702,
+      "mean_token_accuracy": 0.8480874240398407,
+      "num_tokens": 2773117.0,
+      "step": 385
+    },
+    {
+      "entropy": 0.5497441381216049,
+      "epoch": 2.9545454545454546,
+      "grad_norm": 0.78125,
+      "learning_rate": 7.906200651819907e-08,
+      "loss": 0.5609755039215087,
+      "mean_token_accuracy": 0.8707708656787873,
+      "num_tokens": 2808716.0,
+      "step": 390
+    },
+    {
+      "entropy": 0.5683890715241432,
+      "epoch": 2.992424242424242,
+      "grad_norm": 0.74609375,
+      "learning_rate": 6.455604043331676e-09,
+      "loss": 0.5778711795806885,
+      "mean_token_accuracy": 0.8663623631000519,
+      "num_tokens": 2849722.0,
+      "step": 395
+    }
+  ],
+  "logging_steps": 5,
+  "max_steps": 396,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 3,
+  "save_steps": 20,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.478377839274039e+17,
+  "train_batch_size": 4,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1320b30f0d3660d9a16861093b02f9cff5211eff22ca115da6ea421ac390673f
+size 5777