Tokenization Isn't Preprocessing — It's Architecture

Tokenization is the first boundary between human-readable text and the sequence an LLM actually computes over. A model does not receive words, sentences, or source code directly. Text passes through a tokenizer that may normalize Unicode, apply pre-tokenization rules, segment the text into learned pieces, map those pieces to integer IDs, and insert special tokens before the model ever sees an embedding. That preprocessing step has consequences across the system. It changes how much text fits into a fixed context window, how efficiently different languages and code are represented, how numbers and identifiers are split, which vocabulary entries receive useful training signal, and even whether metrics such as perplexity can be compared fairly across models. It also explains why several common simplifications are wrong. BPE is not inherently byte-level. SentencePiece is not a single tokenization algorithm. Token IDs are labels tied to learned model parameters, not semantic coordinates. And once a model has been trained around a tokenizer, replacing that tokenizer is not a harmless preprocessing swap. I put together a technical handbook that works through this from first principles => Unicode and UTF-8, normalization, pre-tokenization, BPE, byte-level BPE, Unigram, WordPiece, vocabulary construction, special tokens, chat templates, multilingual and code behavior, numerical tokenization, context length, under-trained tokens, and the failure modes that usually get skipped. The explanations are grounded in the original papers and current implementations. Sharing Understanding Tokenization here:

Tokenization is the first boundary between human-readable text and the sequence a model actually computes over. Text isn't received as words or characters — it passes through a tokenizer that normalizes Unicode, applies pre-tokenization rules, segments text into learned subword pieces, maps those pieces to integer IDs, and inserts special tokens, all before an embedding lookup ever happens.

This has real downstream consequences: how much text fits in a fixed context window, how efficiently different languages and code get represented, how numbers and identifiers get split, which vocabulary entries receive enough training signal to be useful, and whether perplexity can even be compared fairly across models with different tokenizers.

It also corrects a few common simplifications: BPE is not inherently byte-level (that's GPT-2's specific variant). SentencePiece is a framework, not an algorithm — it implements both BPE and Unigram. Token IDs are arbitrary vocabulary indices, not semantic coordinates — meaning lives in the learned embeddings, not the integer. And once a model is trained around a tokenizer, swapping that tokenizer breaks the embedding-vocabulary correspondence — it's not a drop-in preprocessing change.

I removed the announcement framing ("I put together a handbook," "Sharing here") since that's promotional scaffolding, not content — everything technical survived unchanged.