The boring baseline wins

Zafrullah Khan, Ed.D. · about 6 minutes

2026-10-22

Before you train a model or write a clever prompt, build the dumbest system that could work. If it’s good enough, you’re done. If it isn’t, it becomes the bar every fancier system has to clear. Leave this page with one habit and a short recipe you can run on an ordinary laptop.

“Just ask a big chat model”

Daniel and Priya are illustrative examples, not real people. QuietBrew is an illustrative shop, and the bakery friend is an illustrative character.

Monday, 9:04. Daniel, an illustrative shop owner, opens the QuietBrew inbox and finds 61 new messages. A refund that hasn’t arrived. A cracked lid. A café asking about wholesale. Before anyone can answer them, someone has to read each one and decide what it’s about. That takes him most of an hour.

“Just ask a big chat model,” says the bakery friend. “It understands English.”

Priya asks a better question: “How often is it wrong, how long does it take, and what does each answer cost?”

That question is the whole lesson. Here is how Chapter 17 answers it with numbers.

Write the spec first, including a NEVER line

Sorting messages by what the customer wants is called intent classification: read a message, pick one label from a fixed list. To give the contest real data, the chapter rehearses on BANKING77, a public dataset of 13,083 customer-service questions for an online bank, each labeled with one of 77 intents. It was released by PolyAI with a 2020 paper by Casanueva and colleagues under a CC BY 4.0 license (full credit at the end of this page).

The spec set three rules before any code ran:

NEVER  Return a label that isn't on the list.
DONE   At least 90% accuracy on 3,080 held-out messages,
       a valid label 100% of the time, and under 100 ms
       per message on a shop laptop CPU.
COST   No per-message fee. Retraining must fit in an
       afternoon when the labels change.

The NEVER line matters because routing software can’t handle a label it doesn’t know. A person reads “refund_issue” and understands it. A routing table drops the message on the floor. So an invalid label gets counted separately from a wrong one.

The boring baseline: no language model at all

System 1 turns each message into a list of numbers with a small, open embedding model (all-MiniLM-L6-v2, about 22 million parameters), so messages that mean similar things land near each other. Then it hands those numbers to logistic regression, one of the oldest tools in machine learning, which learns which label each region belongs to.

The core of it fits on one screen (adapted from Example 17.2 of the book; MIT License):

from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression

m = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
Etr = m.encode(train_texts, normalize_embeddings=True)
Ete = m.encode(test_texts, normalize_embeddings=True)

clf = LogisticRegression(max_iter=2000, C=10).fit(Etr, train_labels)
pred = clf.predict(Ete)

On the book’s 8-core test machine, with no GPU:

92.9% accuracy on all 3,080 test messages, past the 90% bar.
12 ms per message, one at a time.
Under two minutes to train.
Zero invalid labels. Logistic regression can only pick from the labels it was trained on.

The fancy options

System 2 fine-tuned a small language model, SmolLM2-135M, with a classification head: a layer that gives one score per label, so its answer is always one of the 77. It scored 93.1% at 54 ms per message, after 30 to 50 minutes of training. That is six more messages right out of 3,080. A paired comparison of the two systems on the same messages called it a tie.

System 3 was the bakery friend’s idea: no training, just put the 77 labels in a prompt and ask a 0.8-billion-parameter chat model (Qwen3.5-0.8B) to pick one. On a 308-message slice of the test set it scored 47.1%, at 1.8 seconds per message. And 14 of its replies, 4.5%, weren’t labels at all. Some said “yes”: the model seems to have answered the customer instead of sorting the message. Others invented labels that sound right and don’t exist, like “card_declined”.

Read the scorecard honestly

System	Accuracy	Invalid labels	Time per message	Training
Embeddings + logistic regression	92.9%	0	12 ms	About 2 minutes
Fine-tuned SmolLM2-135M	93.1%	0	54 ms	30 to 50 minutes
Qwen3.5-0.8B, prompted	47.1% (slice)	4.5% (slice)	1,800 ms	None

Our test, not a general ranking. Rows 1 and 2 were scored on BANKING77’s 3,080-message test set; row 3 on a 308-message stratified slice (4 per intent). All three ran one message at a time on the book’s 8-core CPU test machine, with no GPU, using the versions pinned in the book and measured for its October 2026 edition. The chat model got the 77 label names in a zero-shot prompt, with thinking mode off and greedy decoding (not the sampling settings its model card recommends), and we kept the first word of each reply. It was not trained on this task. The scripts are on the companion page. Your results may differ with other hardware, versions, prompts or settings.

This doesn’t say bigger is worse. The chat model was never trained on this job; it was only prompted. The narrower, more useful lesson: when the output is a fixed list, give the model a fixed list. In this test, both trained classifiers beat the prompt, and a small one was enough.

Speed is a cost, too. At 1.8 seconds per message, one at a time, our test machine sorts about 2,000 messages an hour. At 12 milliseconds, about 300,000. If you pay per call, that difference is the bill.

Ship the boring one, and keep the fancy one

Systems 1 and 2 both pass every line of the spec, and their accuracy is a tie. So the tie-breakers are the things the spec cares about next: speed, retraining time, and how many moving parts can break. The boring baseline wins all three. Daniel ships it.

The fine-tune isn’t wasted. It’s the measured answer to “would a language model do better?”, and without it he’d wonder forever.

Labels are the expensive part, so it helps to know how many you need. Retrained on smaller samples, the baseline reached 84.8% with 10 examples per intent and 88.6% with 20. Adding a new intent, like “wholesale”, costs roughly an hour of reading old email to write about 20 good examples, and two minutes of compute.

Try it this week

Pick a sorting job you already do by hand: tickets, emails, form replies.
Write a three-line spec: the label list, the NEVER rule, and what DONE means (accuracy and speed).
Write about 20 example messages for each label. A day later, write 10 new ones per label and keep them apart as your test.
Run the recipe above and score it on those new messages.
Only if it misses your DONE line, try something bigger, and make it beat this number.

If the boring one meets the spec, you’re done, and you can spend your time on better data. If it doesn’t, you now know exactly how much the expensive system has to add to be worth it.

From Chapter 17, Capstone: A Small, Fast Classifier and Extractor, of Train Your Own LLM (external) by Zafrullah Khan, Ed.D. The code above is adapted from Example 17.2 of Train Your Own LLM by Zafrullah Khan, Ed.D. (ai021.ai, Inc., 2026). The chapter’s full scripts are on the companion page under the MIT License.

Data: BANKING77 (external) by PolyAI (Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson and Ivan Vulić, “Efficient Intent Detection with Dual Sentence Encoders (external),” NLP4ConvAI 2020), licensed under CC BY 4.0 (external). We used the published train/test split without changes and scored the prompted model on a 308-message stratified slice of the test set.

Models: all-MiniLM-L6-v2 (sentence-transformers), SmolLM2-135M (Hugging Face) and Qwen3.5-0.8B (Qwen) are third-party open models released under the Apache License 2.0. We name them only to say what we tested; their makers have not reviewed or endorsed this article.