Viveka-34M (विवेक)
Viveka is Sanskrit for discernment: telling what is right from what only looks right.
A small English language model trained from scratch (its own tokenizer, random start) on free Colab T4 GPUs. Its strength is judging: given a few versions of a sentence, it tells which one makes the most sense.
Model
| Design | Llama-style (rotary positions, SwiGLU, RMSNorm), 8 layers, width 512, 33.9M numbers |
| Tokenizer | its own byte-level BPE, 16,000 word pieces |
| Context | 256 word pieces |
| Training | 45 minutes from scratch, then 150 more minutes at a lower learning rate (one T4 each) |
| Stored as | half precision, 68 MB |
Training text
| Source | Licence |
|---|---|
| FineWeb-Edu (educational web pages) | ODC-BY |
| Project Gutenberg books | public domain |
| English Wikipedia | CC BY-SA |
| IAM handwriting texts (first 45 minutes only, 10%) | non-commercial research |
Because of the IAM texts, this model is non-commercial (CC BY-NC 4.0).
Results
On 20 fresh everyday sentences that no version trained on (lower is better):
| Perplexity per word | Sense test | |
|---|---|---|
| First Viveka-34M | 233.8 | 12 / 12 |
| This version | 176.9 | 12 / 12 |
The sense test asks it to pick the sensible sentence ("She drank a glass of cold water." over "...cold window.").
Use
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('vrishv/Viveka-34M')
lm = AutoModelForCausalLM.from_pretrained('vrishv/Viveka-34M', dtype='float32')
x = tok('The cat sat on the', return_tensors='pt')
print(tok.decode(lm.generate(**x, max_new_tokens=20, do_sample=True, top_k=40)[0]))
Limitations
Very small: it knows grammar and common phrases but little about the world. It is meant for judging sentences, not chatting or facts. English only; it may repeat biases or old-fashioned language from its training text.
About the maker
Viveka was built by Vrishvardhan Sharma (vrishv), a 9-year-old self-taught builder from India. His other work:
- GitaVardhan: a free, offline Bhagavad Gita app in 13 Indian languages, and a pocket-sized voice-controlled Gita device he designed, wired and coded himself.
- Sanskrit Vardhan: an app for learning Sanskrit through the Bhagavad Gita, with word-by-word meanings and step-by-step Pāṇinian derivations. Launched on 20 October 2026 by Prof. Shrinivasa Varakhedi, Vice-Chancellor of Central Sanskrit University.
- Kannada, Tamil and Telugu Vardhan: apps that teach everyday spoken Kannada, Tamil and Telugu.
- Hindi Vardhan (hindivardhan.in): a dictionary of pure, Sanskrit-rooted Hindi. Over 6,000 borrowed words in everyday Hindi, each with native alternatives checked against five classical Sanskrit dictionaries.
- AksharaVardhan: a handwriting reader that runs on an ordinary computer.
He has memorised the Bhagavad Gita and Pāṇini's Aṣṭādhyāyī, won national Sanskrit recitation competitions, and speaks six languages. More at vrishvardhan.in.
CC BY-NC 4.0 © 2026 Vrishvardhan Sharma.
- Downloads last month
- 60