Week 2: Postcards from AI history: seventy years, one idea

Week 2 · History that clarifies today

Postcards from AI history: seventy years, one idea

Zafrullah Khan, Ed.D. · about 9 minutes

2026-10-15

Understand the machine first. Then train, fine-tune, and deploy honestly.

Seventy years, one idea

People often talk about AI as if it appeared overnight in late 2022. It didn’t.

In 1948, a mathematician at Bell Labs generated fake English from letter counts. By 2017, attention alone powered a new architecture. By 2022, a free chat window made that architecture feel like a new kind of mind. The core task never changed: predict what comes next.

That through-line is the most useful mental model you can carry into a project. Prediction is the engine. Fluency is the trap. Data, post-training, and licenses are where today’s builders win or lose.

Here are six postcards from that history — each with a habit you can use this week.

1948 · Shannon: prediction is the engine

Claude Shannon wanted to understand how much information a message really carries. To make his point, he generated fake English.

He started with letters picked completely at random. Then he picked letters according to how often each one appears in English. Then he picked each letter based on the letter just before it, then the two letters before it. With each step, the fake text crept closer to the real thing. His third-order approximation read:

IN NO IST LAT WHEY CRATICT FROURE BIRS GROCID
PONDENOME OF DEMONS TURES OF THE REPTAGIN IS
REGOACTIONA OF CRE.

Shannon wasn’t building a chatbot. He was demonstrating, with pencil-and-paper statistics, the idea behind every large language model: if you can predict what comes next, you can generate language. The better your predictions, the more real the language looks.

The same idea still runs training today. When you watch a number called loss fall, you are watching the model become less surprised by the actual next token. Shannon gave us the vocabulary of surprise — bits, entropy, information — that training still uses.

Habit: when someone asks what a language model “knows,” answer with what it is guessing from, and how you plan to steer those guesses.

1966 · ELIZA: fluency is the trap

Not every early talking program used statistics. In January 1966, Joseph Weizenbaum published ELIZA in Communications of the ACM. Its best-known script imitated a psychotherapist who mostly reflects your words back at you. Say “I’m worried about my mother,” and ELIZA might reply “Tell me more about your family.”

ELIZA had no statistics and no learning. It spotted keywords and applied hand-written rules that chopped and rearranged your sentence. Weizenbaum was candid about how shallow this was. His paper opens with the observation that “to explain is to explain away”: once a program’s inner workings are laid out plainly, its apparent magic “crumbles away.” Yet many people who chatted with ELIZA attributed real understanding to it. That tendency is now often called the ELIZA effect.

The effect hasn’t gone away. It’s stronger, because today’s models are far more fluent. A reply that sounds thoughtful is the easiest thing for a language model to produce, so it is the weakest evidence that the model is right.

Habit: score answers against a test you wrote in advance, not against how they feel.

2017 · The Transformer: built for translation, not chat

For decades, statistical models hit three walls: most long word sequences never appear in any training text (sparsity), words like “cat” and “dog” get no shared meaning (no similarity), and practical models could only look back a few words (short memory). Neural embeddings helped with similarity. Recurrent networks and LSTMs helped with memory. Then attention let a decoder look back at every input position when translating.

In June 2017, eight researchers — most of them at Google — posted Attention Is All You Need. They proposed the Transformer, “based solely on attention mechanisms,” dispensing with recurrence entirely.

Two reasons that mattered:

1. Parallel training. An RNN must process token 1 before token 2. A transformer processes all positions of a training sequence at once, which suits GPUs. The paper reports a new state-of-the-art English-to-French score after training for 3.5 days on eight GPUs.
2. Short long-range paths. In an RNN, information from the first word has to survive many update steps to reach the fiftieth. In a transformer, the fiftieth word can attend to the first directly.

Nearly every LLM you will work with is a descendant of this design. The transformer was introduced for machine translation, not chat. Chat came later. Architecture was a genuine leap; many gains since have come from data, scale, and training procedure, refining a design that has been broadly stable since 2017.

Habit: when a release announcement sounds revolutionary, ask which part is new architecture, and which part is data or post-training on a familiar design.

2022 · Chinchilla: feed the model

As models grew, researchers asked whether there was a pattern to how much better they get as you make them bigger.

In January 2020, a team at OpenAI found that loss improves in a smooth, predictable way as you increase model size, dataset size, and compute. Their analysis suggested that, for a fixed compute budget, the best strategy was very large models on a relatively modest amount of data.

Two years later, in March 2022, a DeepMind team revisited the question in Training Compute-Optimal Large Language Models. After training more than 400 models, they concluded that many large models of the time were significantly undertrained: too many parameters, not enough data. Model size and training tokens should grow together. Double the parameters, double the training tokens.

To test this, they trained Chinchilla: 70 billion parameters and four times more data, on the same compute budget as their earlier 280-billion-parameter Gopher. Chinchilla outperformed Gopher, and also GPT-3 and other much larger models, across a wide range of tasks. Being smaller, it was also cheaper to fine-tune and run.

Published tokens-per-parameter ratios (from the Chinchilla paper’s Table 1, the Llama 2 paper, and the SmolLM3 model card) make the shift visible:

Since Chinchilla, many open builders have gone far past that ratio on purpose: train a relatively small model on a very large amount of data, because small models are cheaper for everyone who downloads them.

Habit: treat data as a first-class ingredient. A modest model on clean, sufficient data usually beats a bigger one trained on scraps. Try a tiny version of an experiment before you spend real money on a big one.

2022 · InstructGPT and ChatGPT: ingredients, not a new kind of AI

By 2021, large language models were impressive but awkward. Ask GPT-3 a question and it might answer — or continue with a list of similar questions. Good behavior required careful prompt engineering.

In March 2022, OpenAI published Training language models to follow instructions with human feedback, describing InstructGPT. The recipe had three stages:

1. Collect human-written demonstrations of good behavior and fine-tune on them (supervised fine-tuning).
2. Have labelers rank several model outputs from best to worst.
3. Use those rankings to further tune the model with reinforcement learning from human feedback (RLHF).

On the paper’s prompts, people preferred outputs from the 1.3-billion-parameter InstructGPT model over outputs from the 175-billion-parameter GPT-3 base model — despite 100 times fewer parameters. Teaching beat size.

On November 30, 2022, OpenAI released ChatGPT as a free research preview. Its announcement called it a “sibling model to InstructGPT,” trained with the same RLHF methods but with dialogue data. The announcement was also frank: ChatGPT “sometimes writes plausible-sounding but incorrect or nonsensical answers.”

ChatGPT did not introduce a fundamentally new kind of model. Its ingredients — a pretrained transformer, supervised fine-tuning, and preference-based tuning — were all described in earlier public work. What changed was the interface: a free chat window anyone could use.

Habit: remember that post-training is where usefulness gets shaped, and it is the part of the pipeline you can do yourself at small scale. You do not need to pretrain a giant to teach a small model a behavior.

Now · “Open” is four questions

“Open model” is used loosely. When people call a model open, they might mean any of these:

1. Open weights — you can download the trained parameters and run or fine-tune the model yourself.
2. Open code — the code used to train the model is published, so you can see and reproduce the procedure.
3. Open data — the training data, or at least a detailed description of it and how to obtain it, is published.
4. An open license — the legal terms let you use, modify, and share the model for the purposes you need. Common examples include Apache 2.0 and MIT. Some models use custom licenses with extra conditions.

These dimensions vary independently. A model can have downloadable weights but a restrictive license, or a permissive license but no information about its training data.

On October 28, 2024, the Open Source Initiative released version 1.0 of the Open Source AI Definition. Broadly, it asks for the freedoms to use, study, modify, and share, plus access to detailed information about training data, the complete training and inference code, and the model parameters. Not everyone agrees with every detail, but it is a solid checklist.

Licenses can change between generations of the same family. Always check the license of the exact model you download, not just the family name.

Habit: before you build on a model, answer all four questions on its model card. A yes on one tells you nothing about the others.

SmolLM3: a model that shows its work

To make “open” concrete, look at Hugging Face’s SmolLM3 — a case study whose makers published nearly everything (from its public model card and linked resources).

SmolLM3 is a 3-billion-parameter, decoder-only transformer, released under the Apache 2.0 license. It was pretrained on 11.2 trillion tokens with a staged curriculum of web, code, math, and reasoning data whose mix shifted over training. After pretraining came mid-training on reasoning tokens, then supervised fine-tuning, then preference alignment.

What makes it a teaching gift is what was published:

• the weights, both base and instruction-tuned
• the training and evaluation configurations and code
• the pretraining datasets
• intermediate checkpoints, so you can study how the model changed as it learned
• the supporting tools, also open source

Three lessons travel well to any budget:

1. Data is designed, not just collected. A staged curriculum shows how much thought goes into which data the model sees and when.
2. Post-training is layered. Mid-training, supervised fine-tuning, and preference optimization each add something.
3. Openness enables learning. Because the configs and checkpoints are public, you can read exactly how a modern small model was built.

You will not pretrain at that scale — it used hundreds of GPUs. You can use the same kinds of tools and follow the same stages at a size that fits your hardware.

Habit: if you want to learn how models are built, start with one that shows its work. Open its model card and see how many of the four openness questions you can confirm. It takes about ten minutes, and you will read every card differently afterward.

What to take into your next project

Compress the seventy-year story into five working rules:

• Prediction is the engine. From Shannon’s letter tables to today’s transformers, the core task has not changed: predict what comes next.
• Fluency is the trap. ELIZA showed how easily people read understanding into fluent text. Score, don’t feel.
• Architecture matters — then data and training take over. The transformer was a leap. Many later gains came from data, scale, and post-training on a design that has stayed broadly stable.
• Data and size must be balanced. Undertrained giants lose to well-fed smaller models.
• Post-training and licenses decide what you can ship. Usefulness is shaped after pretraining. Openness is four separate checks.

None of this requires a GPU. It requires clearer questions before you touch training code.