SENDERWIRE

A 1.1B parameter language model trained end-to-end from scratch on a small budget, on a single commodity GPU node, by Aniket Kumar (Threatthriver).

SENDERWIRE is a conversational (chat) model. It was pretrained from scratch and then supervised-fine-tuned (SFT); this repository contains the SFT chat checkpoint — the model you actually talk to.

Honest positioning

This is a good model for its compute budget, and that is the whole claim. It will not match models like Qwen3-0.6B or SmolLM2-360M on benchmarks — those were trained on trillions of tokens, versus a few billion here. The gap is training data by orders of magnitude, and no architecture change closes it. What a small budget buys is roughly GPT-2-class capability at a tiny fraction of GPT-2's original training cost. It is not a frontier model and this card will not pretend otherwise. Expect occasional factual mistakes and confident-but-wrong answers, especially about specific people or recent events.

Model details

Parameters	~1.1B (1408 d_model, 22 layers, 11 heads)
Context length	2048 tokens
Vocab size	32768
Attention	Grouped-query capable; sliding-window pattern SSSL
Precision	trained in bf16
Validation bits-per-byte	0.2780 (SFT checkpoint)
Stage	pretrain → SFT (this repo is the SFT checkpoint)
Architecture	nanochat-family GPT: rotary embeddings, QK-norm, untied embeddings, relu² MLP, no biases

Files

File	What it is
model_000467.pt	SFT model weights (PyTorch checkpoint, ~3.2 GiB)
meta_000467.json	Model config + training metadata
tokenizer.pkl	Tokenizer (tiktoken-based, 32768 vocab)
token_bytes.pt	Tokenizer byte table

The optimizer shards used for resuming training are intentionally not included — they are not needed to run the model.

How to use

SENDERWIRE uses the nanochat-family code (the training and inference stack this model was built on). It is not a Hugging Face transformers model and will not load with AutoModel. To run it, use the SENDERWIRE / nanochat codebase, which loads the .pt checkpoint directly.

# Inside the OXPID repository (which ships the model definition and loader):
from nanochat.frugal_loader import load_frugal
import torch

model, meta = load_frugal(
    directory="path/to/this/download",  # folder containing model_000467.pt + meta_000467.json
    step=467,
    device=torch.device("cuda"),
    dtype=torch.bfloat16,
)
model.eval()

The repository also provides a chat server (scripts/serve.py) with a web UI and an OpenAI-compatible API, and a terminal client (scripts/chat_client.py).

Training

• Pretraining: from scratch on an English web-text mixture, ~billions of tokens, single commodity multi-GPU node.
• Supervised fine-tuning: chat/instruction data plus task mixtures (MMLU, GSM8K) to make it conversational.

License

Proprietary — all rights reserved. See LICENSE. The model weights are an original work of Aniket Kumar. The training code SENDERWIRE was built on derives from karpathy/nanochat (MIT); trained weights are a distinct work from the code that produced them, so the weights are licensed independently of that MIT code. If you redistribute the source tree, retain nanochat's MIT notice; distributing these weights is governed by this repository's license.

Citation

@misc{senderwire2026,
  title  = {SENDERWIRE: a 1.1B language model trained from scratch on a budget},
  author = {Kumar, Aniket},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/Threatthriver/SENDERWIRE}}
}

Acknowledgements

Built on Andrej Karpathy's nanochat.