Write the test first

Zafrullah Khan, Ed.D. · about 6 minutes

2026-10-16

Most projects that set out to train a model fail on the goal, not the code. The script can finish and the loss can fall, and you can still have trained the wrong thing. Leave this page with two habits you can use the same day: a 12-line model spec, and a starter eval of 10 examples written before you train.

I want the model to know my documents

That wish sounds practical. It usually names the wrong job.

Fine-tuning teaches behavior. A house voice, a fixed way of answering, a format you want every time: those are behaviors. Retrieval changes what the model can look at. A manual, a policy, or a thread the model was never shown is something you hand over when the question is asked. You do not bake a snapshot of those files into the weights and hope the snapshot stays true.

“Know my documents” is almost always a retrieval gap. If the real gap is how the answer should sound or how it should be shaped, that is behavior, and it comes later. It comes after you can say what good is, and after a careful prompt has had a fair try. Fine-tuning is a poor way to pour a folder of files into a model.

Two weekends

Both weekends below are illustrative. They are a contrast, not a report. Priya is an illustrative example, not a real person. Daniel is an illustrative example, not a real person. QuietBrew is an illustrative shop, not a real business.

In the first illustrative weekend, Priya, an illustrative example, trains first. She uses about 4,000 old emails. The answers sound like the shop. One of them quotes a customer address. On Monday, Priya, an illustrative example, deletes the notebook.

In the second illustrative weekend, Daniel, an illustrative example, writes one sentence and the success criteria before any training. He labels 100 emails as a held-out test: examples he will score, and will not train on. He checks the license. He strips personal information. He tries a careful prompt and writes down the score. Then he fine-tunes and scores those same emails again. QuietBrew, the illustrative shop in that second weekend, never becomes a case study. It only keeps the two orders of work apart.

The illustrative weekend that Priya, an illustrative example, spent produced a model she had to throw away. The illustrative weekend that Daniel, an illustrative example, spent produced a comparison: the prompt, then the fine-tune, on mail neither run was allowed to learn from.

Three questions

Before you open a training notebook, use a three-question gut check. The first question holds the other two: can a prompt do it, and can you measure it?

Can you state the task in one sentence? A sentence you could hand to someone else. “Answer a customer email in our voice, and do not invent an address” is a task. “Know our documents” is a wish. If the sentence needs a paragraph of exceptions, the project is not one project yet.

Can you score the outputs? You need a way to mark an answer acceptable or not, or at least better or worse than another answer. If you cannot score it, you cannot tell whether training helped, and you cannot tell whether the prompt was already enough.

When both answers are yes, you have something you could train. When either answer is no, you are not ready, and more code will not fix that.

The 12-line model spec

A spec with blank lines is a map of what you don’t know yet, which is extremely useful in itself.

MODEL SPEC: <project name>          Date: <date>

1. Task (one sentence):
2. Users and setting (who uses it, where it runs):
3. Input -> output (with one real example of each):
4. Success metric and target:
5. Baseline (prompting result on the eval set):
6. Evaluation set (size, where it came from):
7. Base model (name, size, license):
8. Training data (source, size, rights, privacy):
9. Method (prompting / retrieval / SFT / LoRA /
   QLoRA / preference tuning / from scratch):
10. Budget (hardware, hours, money):
11. Risks and safeguards:
12. Kill criteria (when to stop):

The blank form is free. There is nothing to sign up for. Download the blank spec as a text file and the same blank as a one-page PDF.

Write the eval set first

A teacher who writes the exam after seeing the student’s answers is no longer giving an exam. Write the test first. Then train, or decide not to.

Plan on 50 to 200 held-out examples: inputs the training data never includes, each with the answer you would accept. You do not need the full set this afternoon. Start with 10 today, and include 2 hard cases in those 10. A hard case is one your careful prompt already gets wrong. Another easy note that any draft would pass does not teach you where the project breaks.

Then don’t fix the test. When a score is disappointing, change the prompt or the model. Leave the examples where they are. If a label was simply wrong, correct it where someone else can see the change, and say why. Do not quietly edit the set so this week’s number looks kinder.

Decide when you will stop

Write the kill criteria before you start. Name the score, on that held-out set, at which you will drop the training idea and keep what you already measured. “Ship the prompt” is a valid win. It means the careful prompt already met the one-sentence task, and you have the score to show it. A fine-tune earns the next week only when it beats that line by enough to justify the license check, the personal information you removed, and the time. If it does not clear the line, the decision is already made. You do not owe the project a training run.

From Chapter 3 of Train Your Own LLM.