Demystifying LLMs for Healthcare: A Fact-Check of Stanford's Popular Explainer

Generative AI for Healthcare (Part 1): Demystifying Large Language Models
This video lecture from Stanford covers
- Introductions and Disclosures
- Why Is Prompting Hard?
- The Three Epochs of Healthcare AI
- Tokenization and Embeddings
- Transformer Architecture and Self-Attention
- Pre-Training and the Evolution of LLMs
- Post-Training: Making the Model Helpful and Aligned
- The Reasoning Era: Scaling Test-Time Compute
Lecture - https://lnkd.in/d7b27wMw

Confirmed accurate. Now let's check Sam Altman's tweet on GPT-4.5 being the last non-reasoning model, and Ilya Sutskever's NeurIPS 2024 comment.I found the full transcript (via search, since the LinkedIn short-link and YouTube page aren't directly fetchable) and checked the specific factual claims made in the lecture. Here's what I found — overall the lecture is well-researched and highly accurate, with a few small numerical slips.

Confirmed accurate:

• "Attention Is All You Need" (2017) — correctly attributed to the Google Brain team, introduced the transformer/self-attention architecture.
• GPT-1 (June 2018) — 117M parameters, trained on BooksCorpus (~7,000 unpublished books), 12-layer decoder-only transformer. Matches the original OpenAI paper.
• GPT-2 (2019) — 1.5B parameters (a ~13x jump), trained on WebText, 40GB of text scraped from URLs linked in Reddit posts with 3+ upvotes, ~8 million web pages. All matches OpenAI's own writeup.
• "Scaling Laws for Neural Language Models" (Jan 2020) — correctly identified as the pivotal OpenAI paper on jointly scaling compute/data/parameters, and DeepMind's Chinchilla paper is correctly noted as later complicating (not overturning) it.
• GPT-3 (June 2020) — 175B parameters (a 116x jump from GPT-2) — correct. Compute figure of ~3,600 petaflop/s-days matches the commonly cited 3,640 PF-days from the GPT-3 paper.
• GPT-3.5/ChatGPT (Nov 2022) and the RLHF/SFT explanation — conceptually accurate.
• Constitutional AI / "LLM-as-a-judge" — correctly attributed to Anthropic.
• Jensen Huang's CES 2025 keynote on "three scaling laws" (pre-training, post-training, test-time) — confirmed accurate via multiple reports of the actual keynote.
• Sam Altman's tweet that GPT-4.5 would be OpenAI's "last non-chain-of-thought model" — confirmed verbatim from his Feb 12, 2025 post on X.
• Ilya Sutskever's NeurIPS 2024 remarks that pre-training as we know it will end due to finite high-quality data — this happened at NeurIPS 2024 (the transcript/summary I found says "Europe's 2024," which looks like a mis-transcription/mishearing of "NeurIPS," not an actual factual error in the lecture itself).

Minor inaccuracies worth flagging:

• GPT-3 training tokens: The lecture states "roughly 400 billion tokens." The actual GPT-3 paper reports the model was trained on 300 billion tokens (drawn from a mixture of datasets whose raw sizes summed higher, with filtered Common Crawl alone at ~410B tokens). So "400 billion" isn't the standard figure for total training tokens — it's off by roughly 25–35%, likely conflating the filtered-CommonCrawl-only figure with the full training token count.
• GPT-3 hardware: The lecture says training used "1,000 Nvidia A100 GPUs." The original GPT-3 paper says training ran on V100 GPUs on a Microsoft-provided cluster — A100s weren't released until mid-2020, right around/after GPT-3's training, so this detail is likely an anachronistic simplification.
• GPT-2 compute (~600 PF-days): The speakers themselves flag this as their own estimate, since OpenAI never published exact GPT-2 training compute — this is an appropriately caveated approximation, not a hard error.

Not independently verifiable / reasonable estimates presented as such:

• GPT-4's ~$100M training cost, ~3,500GB parameter storage, and ~7,200 MWh energy use are widely-cited industry estimates (OpenAI hasn't officially confirmed these), and the lecture doesn't overstate their certainty.
• The "100 trillion GB" figure for humanity's 2025 data output is a rough, illustrative estimate rather than a precise, sourced statistic — treat it as an order-of-magnitude illustration, not a hard fact.

Bottom line: This is a technically careful lecture; the core architecture/training narrative (GPT-1→GPT-4, scaling laws, RLHF, test-time compute) all checks out against primary sources. The main correction is the GPT-3 token count (300B, not ~400B) and the A100-vs-V100 hardware detail.