The Lottery Ticket Hypothesis: Sparse Networks That Actually Train

Every so often you read a paper that quietly rearranges an assumption you did not know you were carrying. For me, the Lottery Ticket Hypothesis is one of those. It is not new — Frankle and Carbin published it at ICLR in 2019 — but it keeps coming back to mind whenever someone asks why their model has to be so large.

Here is the thing that never sat right with me. You train a big network, you prune 90% of its weights, and the pruned model does fine. Faster, smaller, cheaper. So why not just build the small one to begin with? The standard answer was that it doesn't work: train a small network from scratch and it learns badly. Everyone accepted that and moved on. Big models, the reasoning went, are simply what learning requires.

Frankle and Carbin took the obvious follow-up question seriously and found something stranger.

The part that surprised me

Their claim is that a randomly initialized network already contains a small subnetwork — the winning ticket — that, trained on its own, matches the full model. The catch is in the word "already." The ticket is not something training creates. It is there from initialization, decided by the dice roll of the random weights, and training merely finds it.

The procedure that gets you there is almost annoyingly simple. Train the big network. Prune the smallest weights. Then take what's left and reset those weights to the exact values they had before training ever started, and train again.

That last step is where it stops being a pruning trick. Reset the surviving weights to new random values instead of the original ones — same architecture, same sparsity, same everything but the specific numbers — and it falls apart. Slower to learn, worse accuracy. The structure alone isn't enough. Neither are lucky numbers alone. What works is the pairing: those particular weights sitting in those particular positions.

I find that genuinely odd, in a good way. It means the thing doing the work was never the 100 million parameters. It was a handful of them that happened to start in a good place, and the other 99 million were the price of the search.

What the numbers look like

The results are consistent enough to be uncomfortable. Across the models they tested, tickets land somewhere around 10-20% of the original size, reach the target accuracy two to three times faster, and often generalize slightly better than the network they came out of. Some go much further — down to a few percent of the original parameter count and still training fine.

Large models are less cooperative. At standard learning rates the method stops finding tickets; drop the learning rate, or warm it up gradually, and they reappear. Which is its own small hint that whatever makes a ticket work is fragile early in training, and easy to stomp on if you move too fast.

Why it reframes the question

The framing I keep returning to is that scale buys you lottery tickets. You are not training a big model because every parameter earns its keep. You are training it because more parameters means more independent draws, and more draws means a better chance that one of them started somewhere trainable. Overparameterization isn't capacity. It's search.

That reading also explains the generalization result without much hand-waving. The ticket is smaller than the network that contained it and does better on unseen data, which is roughly what you'd expect if the extra parameters were scaffolding rather than substance.

There is an echo of this in how frontier models are built now. Mixture-of-Experts architectures route each token through a small slice of a very large parameter count — Korea's recent models activate somewhere between 6% and 10% of their total weights at a time. The mechanism is different: MoE picks a different subset per input, while a winning ticket is one fixed subset for everything. But the underlying bet is the same one, that the useful computation is sparse and the bulk exists to make finding it possible.

The catch, and why it still matters in practice

The honest limitation is that you have to train the full model to find the ticket, which means the hypothesis does not — yet — make training cheaper. It makes inference cheaper, which for anything running in production is usually where the money actually goes. Train once, prune hard, deploy the ticket, keep the accuracy.

The open question, the one that would change everything if someone answers it, is whether tickets can be identified before training rather than after. If you could read the initialization and know where the winner is, you would skip the expensive part entirely. Nobody has done that convincingly. A lot of people are trying.

Until then, I think the value of this paper is mostly in how it makes you think about your own models. The next time one feels too big, it is worth sitting with the possibility that it isn't big because the problem is hard. It's big because you're still buying tickets.

---

The paper is The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks by Jonathan Frankle and Michael Carbin (MIT CSAIL, ICLR 2019). It is unusually readable for a research paper and the experiments are laid out clearly enough to reproduce. Worth an hour of your evening.