TinyGPT

Start of the text

Starting…

Show tokens

Decoding settings:

Temperature 0 always takes the likeliest token, and soon repeats itself Top-k keep the k likeliest tokens; 0 turns it off Top-p keep the likeliest tokens that add up to p; 1 turns it off Min-p drop tokens less likely than min-p × the best one; 0 turns it off Repetition penalty push down tokens already written; 1 turns it off New tokens Seed the same seed and settings write the same text

TinyGPT writes the way its training text reads — Hindi Wikipedia and Hinglish — but it learned the language, not the facts. Asked for the capitals of 20 states and countries, it puts the right city first only twice, no better than chance. Treat names, dates and places as invented.

TinyGPT is a decoder-only Transformer — the Llama recipe: rotary positions, RMSNorm, SwiGLU — trained from scratch on 18 million tokens of Hindi Wikipedia and Hinglish, on a laptop. Everything is written by hand: the tokenizer, the model, training, decoding, fine-tuning with LoRA, and the service.

How this page works

Your browser downloads the model files once (55 MB each, kept for your next visit) and does all the work itself: the tokenizer, the forward pass through every layer, the sampling. That code is a JavaScript copy of the Python version, checked against PyTorch: the same token ids, logits within a few millionths, and the same parses.

Parameters	13.8 million
Bits per byte on unseen text (lower is better)	–
Capitals of 20 places: right city ranked first	2 of 20 (chance: 1)
Parser: whole parse right on unseen requests	–
Parser: intent right	–

The parser is a twin of this model — pretrained the same way, minus Hinglish-TOP's validation and test requests — fine-tuned on 2,993 Hinglish-TOP requests. It writes the request back with brackets around the details. The code and a write-up of every step are on GitHub.