The Lottery Ticket Hypothesis: Sparse Networks That Actually Train

The Mystery: Why Do We Need Big AI Models?

Imagine you built a massive AI model with millions of pieces (called "parameters"). After training it, researchers discovered something surprising: you can delete 90% of those pieces and the AI still works just as well.

The smaller version runs faster, uses less memory, and costs less to run. Perfect, right?

So here's the obvious question: why not just build the smaller version from the beginning?

The frustrating answer has always been: because it doesn't work. If you try to train a small AI from scratch, it performs terribly. For years, everyone believed you needed the big, bloated model to learn effectively.

Until two MIT researchers discovered why—and proved that belief wrong.

The Lottery Ticket Hypothesis

In their 2019 research paper, Jonathan Frankle and Michael Carbin from MIT discovered the Lottery Ticket Hypothesis:

When you create a large AI model with random starting values, it contains small sub-models (winning tickets) hidden inside that—when trained by themselves—can perform just as well as the full model.

Think of it like a lottery: When you start a big AI model, you're essentially buying millions of lottery tickets. Most of them are losers. But hidden somewhere in there are winning tickets—small sub-models that got lucky with their starting values and are naturally good at learning.

The key insight: It's not about having a small model. It's about finding the small model that got lucky with the right starting values.

In simple terms: Imagine a classroom of 100 students (the big model). Only 10 of them will become excellent at math, but you don't know which 10 until you start teaching. The "winning ticket" is identifying those 10 students early and focusing all your teaching resources on them.

How to Find a Winning Ticket

The process is surprisingly simple:

• Start with random values - Create a big AI model with millions of random starting numbers
• Train it fully - Let the AI learn from data until it's fully trained
• Remove the weak parts - Delete the pieces that contribute the least (the smallest numbers)
• Reset to the original lucky numbers - Here's the magic: take the remaining pieces and reset them back to their original random starting values

The magic happens in step 4. Instead of keeping the values from training (traditional approach) or using new random values (which fails), you go back to those original lucky starting numbers. Those specific starting values, combined with that specific small structure, create an AI that trains effectively.

Using our classroom analogy: You teach the whole class, identify the top 10 students, then "reset" them by having them retake the course—but this time they're starting with the same natural talents they had originally, and now you know they're the winners worth focusing on.

The Results Are Mind-Blowing

Test Case 1: Simple AI Models

The researchers tested with a model that had 266,000 pieces:

• Old way (doesn't work): Build a small model with random values → train it → performs poorly
• New way (winning ticket): Find the winning ticket → train it → works just as well or better than the full model

Winning tickets at just 21% of original size:

• Train 38% faster than the full model
• Actually perform slightly better (more accurate)
• Can work even when shrunk down to just 3.6% of original size

Translation: Imagine you only needed 21 employees instead of 100, they work 38% faster, and they actually do a better job. That's what's happening here.

Test Case 2: Complex AI Models (Image Recognition)

The same pattern worked across different types of AI:

Model Type 1 (4.3 million pieces):

• At just 8.8% of original size: trains 3.5x faster and performs better
• Can shrink to 2% of original size and still work

Model Type 2 (2.4 million pieces):

• At 9.2% of original size: trains 3.5x faster and performs better

Model Type 3 (1.7 million pieces):

• At 15.1% of original size: trains 2.5x faster and performs better

The pattern is clear: Across all types of AI models, winning tickets are 10-20% of the original size, train 2-3x faster, and often perform better.

Why Using New Random Numbers Fails

Here's the proof that this really works: Take a winning ticket's small structure, but give it brand new random starting numbers. What happens?

It fails completely.

The model with new random numbers learns slower and performs worse—just like the old broken approach. This proves that:

1. Having a small structure isn't enough - you need the lucky starting numbers
2. Having lucky numbers alone isn't enough - you need the right small structure
3. The combination is magic - specific lucky starting numbers paired with the specific small structure

When you give new random numbers to a 21% winning ticket:

• Takes 2.5x longer to learn
• Performs worse (lower accuracy)

The winning ticket's power comes from those lucky starting numbers that—when combined with the right connections—create a model that learns efficiently.

Back to our classroom analogy: If you take those 10 talented students but wipe their memories and give them different natural abilities, they won't perform as well. It's the combination of the specific students AND their specific natural talents that makes them winners.

The Challenge with Very Large AI Models

When researchers tried this on really big, complex AI models, they hit a problem. Using the standard training speed (called "learning rate"), they couldn't find winning tickets—the approach stopped working.

But they discovered a fix: slow down the training process, and winning tickets appeared again. Even better, use gradual speed-up (start slow and gradually increase training speed), and you can find winning tickets even at normal speeds.

Example 1 - Very Large Model (20 million pieces):

• With gradual speed-up: winning tickets work down to 1.5% of original size
• Without it: the method fails to find winning tickets

Example 2 - Complex Model (271,000 pieces):

• With gradual speed-up: winning tickets at 27% of size work perfectly
• Can shrink down to 12% of original size

What this means: For very large, complex AI models, the learning process is more delicate. You need to be gentler to find and train winning tickets successfully.

Classroom analogy: Teaching advanced calculus to gifted students requires a more gradual, careful approach than teaching basic math. Rush it, and even talented students will struggle.

Why This Changes Everything

We've Been Thinking About AI Training All Wrong

This research suggests that the learning process finds and focuses on the lucky pieces. Big models aren't needed because we use all the pieces—they're needed because the more pieces you start with, the more likely you'll have winners hiding inside.

It's a numbers game: buy more lottery tickets (start with more AI pieces), and you're more likely to have a winner.

We Can Train AI Much Faster

If we can identify winning tickets early (or design AI that's more likely to contain them), we could:

• Train small models from the start instead of big ones
• Cut training time and costs dramatically
• Make AI research accessible to more people (not just big tech companies)

The dream: Find the winning ticket before training even starts, then train only that small model. This would make AI training 3-5x faster and much cheaper.

We Can Design Better AI

Winning tickets show us specific combinations that naturally learn well. By studying them, we can design:

• Better ways to set starting values
• More efficient AI structures
• Models that are naturally easier to train

The patterns in winning tickets might teach us core principles about what makes AI trainable.

Reuse Winning Tickets Across Projects

An exciting possibility: winning tickets found for one task might work for related tasks. If a winning ticket captures something fundamental, you might:

• Find a winning ticket once
• Reuse it (or variations) across multiple projects
• Save massive amounts of computation and money

Business impact: Train once, deploy everywhere (in related domains).

Understanding Why AI Works

The lottery ticket hypothesis answers fundamental questions:

Why do AI models work well on new data? Winning tickets are smaller yet work better on new data. This supports the idea that simpler is better—less is more.

Why do big models train better? Not because we need all those pieces, but because bigger = more lottery tickets = higher chance of winners.

Why do starting values matter so much? The specific starting numbers determine which small models can learn effectively. It's not just about random variation—specific numbers create specific opportunities.

Connection to Modern AI (Mixture of Experts)

There's a fascinating connection to Korea's recent AI models which use a "Mixture of Experts" approach. These models have many pieces but only use a small subset at a time:

• A.X K1: 519 billion total pieces, but only uses 33 billion at once (6.4%)
• VAETKI: 112 billion total, uses 10-11 billion (~9%)
• K-EXAONE: 236 billion total, uses 23 billion (9.7%)

This is similar in spirit to winning tickets: you don't need all the pieces active all the time. The key difference:

• Winning tickets: Find one small subset that works for everything
• Mixture of Experts: Dynamically choose different small subsets for each specific task

Both approaches challenge the "bigger is always better" mindset. Both prioritize efficiency. Both prove that the right small subset is more powerful than using everything.

Real-world analogy: You don't need every tool in your toolbox for every job. Winning tickets are like finding the perfect 10-tool kit that handles all jobs. Mixture of Experts is like having a big toolbox but only pulling out the specific tools each job needs.

How You Can Use This Today

While finding winning tickets requires training multiple times, the insights are immediately useful:

Making AI Cheaper to Run

Companies using AI can:

• Train a big model normally
• Find the winning ticket (identify the lucky 10-20%)
• Deploy only the winning ticket (10-20% of original size)
• Get the same accuracy with much lower costs

The winning ticket runs faster, uses less memory, and costs less to operate—but works just as well.

Business impact: If you're running AI in production, this could cut your cloud computing costs by 80-90%.

Understanding What Your AI Actually Uses

Analyzing which pieces survive reveals:

• Which connections are actually important
• Which parts of your AI do the heavy lifting
• Which parts are unnecessary bloat

This helps you design better AI models in the future.

Translation: It's like discovering that only 10 features in your app drive 90% of user value—now you know where to focus.

Better Training Strategies

The research reveals practical training tips:

• Slow down for complex models - Lower training speeds work better for finding winning tickets in large AI
• Use gradual speed-up - Start slow and gradually increase speed for best results
• Train-prune-reset multiple times - Doing this iteratively finds smaller winning tickets than doing it once

These insights help you train more efficient AI from the start.

The Big Unanswered Questions

This discovery raises as many questions as it answers:

Can we find winning tickets without training first? Right now, you need to train the full big model to identify the winners. Can we predict winning tickets just from the starting random numbers?

Can we reuse winning tickets across projects? If you find a winning ticket for one image recognition task, will it work for another similar task? This could save massive amounts of time and money.

What makes starting numbers "lucky"? Winning ticket starting numbers have certain patterns. Can we design better ways to generate starting numbers that are more likely to produce winners?

Why does gradual speed-up help? For large models, you need to start training slow and gradually speed up. Why? What does this tell us about how AI learns?

Can we train winning tickets directly? Can we create training methods that specifically search for winning tickets from the start, instead of finding them afterward by removing pieces?

The Holy Grail: If we could answer these questions, we could train AI models 10-20x faster and cheaper than we do today.

The Bottom Line

The lottery ticket hypothesis completely changes how we think about AI:

Old belief: You need huge AI models for training to work. You can only make them smaller after training for cost savings.

New understanding: Big AI models contain small, efficient sub-models with lucky starting numbers hiding inside. We train big models because they're more likely to contain winners, not because we need all those pieces.

This shifts the key question from "how do we train small AI?" to "how do we find the right small AI with the right lucky starting numbers?"

The practical impact is massive:

• Use 10-20% of the original size with the same performance
• Train 2-3.5x faster in many cases
• Actually works better (higher accuracy on new data)
• Cut deployment costs by 80-90% (less memory, faster processing)

And this is just the beginning. This research opens entirely new directions in building efficient, affordable AI.

The big picture: You don't need massive AI models. You need to find the right small model with the right lucky numbers. That's the real breakthrough.

Key Takeaways (Simple Version)

• Big AI models hide small efficient models inside them with lucky starting numbers
• These small models (winning tickets) work just as well as the full big model when trained with their original lucky numbers
• Both the structure AND the lucky numbers matter - same small structure with different random numbers fails
• Winning tickets are typically 10-20% of original size across different types of AI
• Training speed and gradual speed-up are crucial for finding winning tickets in very large AI models
• Winning tickets train up to 3.5x faster and often work better (higher accuracy)
• Bigger models train better because they contain more potential winners - it's a numbers game
• Repeating the process multiple times finds even smaller winners than doing it just once

What This Means for You

If you're training AI models:

• Try the train-prune-reset process multiple times to find your winning ticket
• Use slower training speeds and gradual speed-up for large complex models
• Deploy the winning ticket instead of the full model to cut costs by 80-90%

If you're doing AI research:

• Study what makes certain starting numbers "lucky"
• Test if winning tickets from one project work on similar projects
• Try to develop ways to identify winning tickets without training the full model first

If you're building AI-powered products:

• Plan to find and deploy winning tickets as part of your standard workflow
• Budget for training multiple times to find the optimal small model
• Think about total cost (training + running in production), not just training cost

The bottom line: Efficient, high-performance AI doesn't require massive models. It requires finding the right small model with the right lucky starting numbers.

And that changes everything about how we build AI.

The Original Research Paper

Title: The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Authors: Jonathan Frankle and Michael Carbin (MIT Computer Science & Artificial Intelligence Lab)

Published: 2019 at the International Conference on Learning Representations (ICLR)

Link: arXiv:1803.03635

The paper is surprisingly readable for a research paper and includes extensive experiments testing the hypothesis across different types of AI models, training methods, and settings. If you're interested in AI training, making models smaller, or efficiency, it's definitely worth reading the full paper.

What makes it special: The authors don't just present an idea—they rigorously test it across multiple scenarios and provide clear, reproducible methods. This is why it's become one of the most influential AI research papers in recent years.

References

• Frankle, J., & Carbin, M. (2019). The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. ICLR 2019.
• LeCun, Y., Denker, J. S., & Solla, S. A. (1990). Optimal brain damage. NIPS.
• Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. NIPS.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR.

---

The lottery ticket hypothesis represents a fundamental shift in how we understand AI training. By revealing that big models contain small, efficient sub-models with lucky starting numbers, it opens the door to dramatically more efficient AI. The winning tickets were hiding inside all along—we just had to learn how to find them.