Train Your Own LLM From Scratch: A Full PyTorch Pipeline From The Pile to RLHF

Grounded / Real Inflated / Uruttu

90% real

10% uruttu

article Original Content

Train your own LLM from scratch!

A step-by-step repo that walks you through building and training a transformer model from scratch using PyTorch. From downloading training data all the way to generating text.

The architecture is built from the ground up following the original "Attention is All You Need" paper. MLP, single head attention, multi-head attention, transformer blocks, and the full transformer model - all coded and explained with detailed diagrams at each step.

Training data comes from The Pile - a diverse 825GB open-source dataset covering books, articles, code, websites, and more. The repo includes scripts to download it, preprocess and tokenize it using tiktoken, store it in HDF5 format, and feed it into training batches.

You can train a 13M parameter model on a single Colab T4 GPU. At 13M parameters the model starts generating proper grammar and coherent short sentences. For billion-parameter training you need at least an A100 or RTX 4090. The repo includes a full GPU compatibility table so you know exactly what's possible on your hardware.

Includes a complete SFT and RLHF guide as a separate notebook for taking your trained model further.

Key capabilities:

• End-to-end pipeline: data download → preprocessing → training → text generation

• Full transformer implementation from scratch with PyTorch

• Trains models from 13M to 2B+ parameters on a single GPU

• Training data from The Pile (825GB, 22 diverse datasets)

• Tokenization via tiktoken (r50k_base)

• SFT and RLHF guide included

100% open source.

I've shared the link in the replies!

verified Validated Content

Train your own LLM from scratch — a step-by-step PyTorch repo covering the full pipeline: raw text → tokenized data → a transformer built from individual components (MLP, attention head, multi-head attention, transformer block, full model) → pretraining → text generation → post-training (SFT, reward modeling, DPO, PPO, GRPO).

Training data comes from The Pile, tokenized with tiktoken's r50k_base and stored in HDF5. A 13M-parameter model trains on a free Colab/Kaggle T4 and produces real words with roughly correct grammar. Larger configs (up to roughly 2B–8B depending on GPU) need cards like an RTX 3090/4090/5090 or A100 — a full compatibility table is included in the repo.

A separate notebook (sft_rlhf_guide.ipynb) walks through SFT and RLHF. The main repo also ships a script-based post-training suite (SFT → reward model → DPO/PPO → GRPO), hand-written in plain PyTorch with no trl, peft, or transformers.

MIT licensed, 10.8k+ GitHub stars.