A small transformer model trained entirely from scratch achieved 44% on ARC-AGI-1, matching leading specialized small models and outperforming many large language models, at a total lifetime compute cost of about $0.67 — covering both training and inference, based on renting a single RTX 5090 for roughly two hours. The model was trained in 1.5 hours and released as fully open source, building on a previous result from the same author that had gone viral and drawn discussion from researchers including Lucas Beyer and Jeremy Howard. The method works by flattening each puzzle's before-and-after grids into token sequences, training a small transformer autoregressively on both the training and evaluation puzzles with the evaluation answers hidden, using per-puzzle learned tags and explicit grid position information rather than treating the grid as flat text. The author's broader thesis is that sample efficiency, not architectural novelty, remains the key unsolved problem in this space, framing the approach as a straightforward application of existing transformer methods with a well-suited data representation.
K
pending
44% on ARC-AGI for 67 Cents: How a Tiny Transformer Beat the Odds
Grounded / Real
Inflated / Uruttu
Original Content
Someone just trained a small AI model and got 44% on ARC-AGI-1 for ~$0.67 🤯
And made the code 100% open-source.
He trained it from scratch in 1.5 hours on one RTX 5090. The $0.67 covers training and inference, which is what a 5090 costs to rent for 2 hours.
ARC-AGI tests whether a model can solve visual logic puzzles it has never seen before. LLMs struggled with it for years. This 8-layer model matches the leading small models built specifically for the benchmark and beats many LLMs.
His previous result went viral and had Lucas Beyer and Jeremy Howard debating it on X. This is the upgrade.
The approach is super simple.
Each puzzle's before-and-after grids get flattened into a sequence, and a small transformer learns from scratch on those sequences, using both the training puzzles and the eval puzzles with answers hidden. Every puzzle gets its own small learned tag so patterns carry across tasks, and the model is told where each cell sits in the grid instead of reading it as a flat string of text.
He stopped making the model predict the puzzle inputs and only trained it on the answers, and the score moved from 40% to 44%. He doesn't know why it helped.
Remove the grid position info or the per-puzzle tags and the score drops to about 24%. Train each puzzle in isolation and it drops to 18%. The representation did most of the work. The other small models credited recursive loops for their scores, and nothing here needed them.
Combining solved puzzles across multiple runs already gets 55%, so he thinks 65% is reachable with a plain transformer and no new ideas.
His take on why nobody did this in 6 years with a million dollar prize on the table: "It's just a transformer with the most obvious representation."
Link to the blog and GitHub repo in the comments.
Validated Content