Week 11: What did your model forget?
Week 11 · Checking a smaller model before you trust it
What did your model forget?
2026-11-03
Smaller and faster is easy to measure. What a model lost only shows up if you go looking.
The habit in one line
When someone says they made the model smaller or faster, ask what it forgot, and whether they re-ran the scorecard.
Every shrinking tool reports two things: how much smaller the model got, and how much faster it runs. None of them reports what it forgot. Only your scorecard does, and only if you run it again.
You don’t need to be an engineer to ask this. You just need four questions, in order.
Step 1: Fit. Does it even need shrinking?
Do one line of arithmetic first: parameters × bytes per weight.
A chess model with about a million parameters, stored the normal 32-bit way, is around 4 MB. That’s smaller than a phone photo. A message router with 22.7 million parameters is about 91 MB, which fits on nearly any device made in the last decade.
If the model already fits, with room left for working memory, you may not need to shrink it at all. Sometimes the best fix is no fix.
Step 2: Equivalence. Does the exported file give the same answers?
Before a model runs on a laptop, a browser, or a phone, it usually gets exported to a new file format. Check that the new file gives the same outputs as the original on the same inputs.
In the book, the exported chess model matched the original with a maximum difference of 0.0000. That’s what you want to see.
This step catches simple mistakes, like a wrong input name or a missing step, before you waste time on accuracy.
Step 3: Score. Re-run the original scorecard on the smaller file
This is the step people skip, and it’s the one that matters most.
Shrinking a model usually means quantizing it: storing each weight with fewer bits, a bit like rounding every price in a shop to a few allowed steps. Most prices barely change. Some blur together.
So take the same test you used to approve the original, and run it on the smaller version. Same questions, same pass mark. If the score drops, you’ve found what it forgot.
Step 4: Device. Try it where it will actually live
Test on the real target: offline, from a cold start, and while other things are running. A page that quietly fetches something from the internet on first load will fail the moment the signal drops.
The simplest version: turn on airplane mode, close the app or browser, open it again, and use it. Until you’ve done that, the honest label is “tested online.”
In the book, the browser version passed this step. The phone steps were written down as a plan and labeled not run, so the book makes no claim about phone results, and neither does this lesson.
The router story
Here’s why step 3 earns its place.
A small model that sorts incoming messages into the right queue scored 92.9% on its test. One line of code made an 8-bit version: 4× smaller and about 2× faster. Every size and speed number looked great.
Then the scorecard was re-run. The first run came back at 79.5%, up to 13 points worse. Roughly one message in seven that used to land in the right place now went somewhere else. Later checks of the same smaller file moved around between 80% and 87%, while the original gave 92.9% every time. The book is honest that the reason for that wobble wasn’t fully tracked down, and the decision didn’t depend on it: a score that moves by several points between runs isn’t a number you can ship against.
The original already fit the device. So the decision was simple: don’t quantize the router. Ship the original.
Shrinking is a tool for models that don’t fit, not a step every model has to go through.
A model in a web page
On-device doesn’t have to mean a phone app.
The same chapter runs the chess model inside a browser with a file called chess.html: 48 lines, no build step, no model server. The page loads the model, turns the moves so far into numbers, and suggests a move.
There’s one more piece, and it’s the important one: a referee. A small chess library lists the legal moves, and the page only plays the model’s best-ranked move from that list. So “never plays an illegal move” holds by design.
It still misses a mate in one in a classic beginner position. Fast and legal isn’t the same as good, so that’s the honest sign to put on it: it runs, it plays legal moves, and it isn’t strong yet.
Your move this week
Pick one model you use, buy, or are evaluating, especially one that’s been made “lighter” or “edge-ready.” Write down four answers:
- Fit: parameters × bytes. Did it need shrinking?
- Equivalence: does the exported or smaller file give the same outputs?
- Score: did anyone re-run the original scorecard on it, more than once?
- Device: has it been tested offline, cold, and under load, or only online?
If one answer is “I don’t know,” that’s your next question for the team. Usually it’s number 3.
Where this comes from
This lesson follows Chapter 20 of Train Your Own LLM by Zafrullah Khan, Ed.D., the hands-on capstone where three small models each get a ship decision. → About the books
Pairs well with an earlier lesson on keeping your own judgment in the work. → Try it first
Want short, practical notes like this in your inbox? → Get the notes