Karpathy's nanochat: The Full LLM Pipeline in One Repo

Karpathy's nanochat repo walks you through the full LLM training pipeline Covers the full LLM pipeline in one place. Tokenization: → BPE tokenizer training from scratch → Bits-per-byte evaluation metrics Pretraining: → Base model training with scaling laws → Single --depth dial sets transformer layers → All other hyperparameters auto-calculate Fine-tuning: → Supervised fine-tuning (SFT) → RL-based alignment → Full chat capability Inference: → CLI chat interface → KV cache optimization Evaluation: → ARC, GSM8K, MMLU, HumanEval benchmarks → DCLM CORE scoring Community speedrun leaderboard: → Challenge: time-to-GPT-2 performance → Current record: 1.65 hours → Open to new submissions Github: https://lnkd.in/d89-ezKR Repost ♻️ for AI developers.

Andrej Karpathy's nanochat repository provides a complete, minimal, from-scratch pipeline for training a ChatGPT-style model, covering tokenization, pretraining, fine-tuning, inference, and evaluation in a single codebase. It includes BPE tokenizer training and bits-per-byte evaluation metrics, base model pretraining guided by scaling laws, and a single --depth parameter that sets the number of transformer layers while automatically calculating all other hyperparameters. The fine-tuning stage includes supervised fine-tuning and RL-based alignment to produce full chat capability, with inference available through a CLI chat interface using KV cache optimization. Evaluation is measured using the DCLM CORE score alongside standard benchmarks. The project also maintains a community speedrun leaderboard tracking the fastest time to reach GPT-2-level performance, with the current record standing at 1.65 hours, and remains open to new submissions.