● A free public tool  ·  Built by Poptani Labs  ·  No sign-up  ·  Your data never leaves your browser
AI, Decoded · 5-minute read

ChatGPT is autocomplete on steroids: how next-token prediction actually works

A glowing artificial neural network diagram layered over a computer chip, illustrating how a language model processes text
Image: 'Artificial Neural Network with Chip.jpg' by mikemacmarketing / photo on flickr, CC BY 2.0, via Wikimedia Commons

This is a research-based explainer: no lab, no hands-on experiments, just the published record of how these systems work. And the record is surprisingly simple. When you ask ChatGPT a question and watch the answer stream out word by word, it looks like the machine thought the whole thing through before typing. It did not. Underneath the fluency, ChatGPT is doing one small job, over and over: guessing the next piece of text.

First, your words get chopped into tokens

Before the model sees anything, your prompt goes through a tokenizer, a fixed program that cuts text into chunks called tokens. A token is usually a whole common word, sometimes part of a word, sometimes a single character or a punctuation mark. The word “unbelievable” might arrive as two tokens, “un” and “believable”; a rare name gets split into pieces the vocabulary already knows. OpenAI’s rule of thumb for English is that one token is roughly four characters, or about three quarters of a word — so 100 tokens is around 75 words. Each token is then swapped for an ID number, and from that point on the model works only with numbers. It never sees your words at all.

The cutting is done by an algorithm called byte-pair encoding, which builds the vocabulary by repeatedly merging the pairs of characters that appear together most often in training text. Common words earn a single token; uncommon ones get assembled from familiar parts. That is also why a model can seem fluent in English and clumsy in other languages — tokenizers are built mostly from the text they were trained on, and English dominates it.

Then the model bets on the next piece

With your prompt converted to token IDs, the model — a neural network called a Transformer, built from a design published by Google researchers in 2017 — looks at the whole sequence and assigns a probability to every token in its vocabulary for what should come next. For a typical vocabulary that is tens of thousands of scores, all adding up to 100 percent. The model picks one, appends it to the text, and runs the whole loop again. An answer of a few hundred words is that loop repeated hundreds of times, a process called autoregressive generation.

How does it choose? The simplest method is greedy decoding: always take the top-ranked token. Real products usually sample instead, picking at random with higher-probability tokens more likely to win, and a setting called temperature controls how flat or sharp the odds are. Low temperature makes the output safe and repetitive; high temperature makes it varied and creative — and more likely to wander off the rails. The model is never checking whether a continuation is true. It is checking whether it is statistically likely, given the patterns in the text it was trained on.

Where the autocomplete comparison breaks down

So is “autocomplete on steroids” fair? Mostly, yes. Your phone keyboard predicts the next word from your last few words; a language model predicts the next token from your entire prompt plus everything it has already written, drawing on patterns learned from trillions of words during training. Predicting the next token well turns out to force the model to learn grammar, facts, code syntax, even rough reasoning — because all of those make the next guess more accurate. That is why a machine trained only to finish sentences can write essays, translate languages and debug code.

But the comparison breaks down in one important way. Autocomplete has no pretence of understanding; a chatbot’s fluency invites us to assume it does. The model has no plan for the answer, no outline, no goal state — it cannot revise a sentence it has already written, and it has no separate fact-checking step. That is why it can state something false with total confidence, and why it struggles with tasks that need real planning or arithmetic carried across many steps. The same mechanism also explains its most famous failure: it invents details when the most likely-sounding continuation is not the true one. That habit has a name — hallucination — and we explain it in detail in Why AI chatbots hallucinate. Image generators work on a different trick entirely, which we cover in How AI image generators actually make pictures.

The time OpenAI called its own model too dangerous

The power of this simple trick became public in an awkward way. In February 2019, OpenAI announced GPT-2, a 1.5-billion-parameter model trained on text from about 8 million web pages, and then refused to release the full version. In a blog post titled “Better language models and their implications”, the lab said it was withholding the trained model over concerns about malicious uses, and would instead release a much smaller model for researchers. The press quickly dubbed it the model “too dangerous to release”.

OpenAI released the model in stages — a smaller 124-million-parameter version in February 2019, larger ones in May and August, and the full 1.5-billion-parameter model only in November 2019 — and reported finding no strong evidence of misuse in the meantime. The episode now reads as a landmark in AI safety policy, but for our purposes it makes a simpler point: the model that worried a leading lab was not doing anything exotic. It was predicting the next token, just far better than anything before it.

What This Means

Knowing that ChatGPT predicts one token at a time changes how you use it. Give it more context and it makes better guesses; ask it to show its working step by step and the extra tokens become a trail it can follow. And when it sounds supremely confident about something you cannot verify, remember what it is actually doing — not looking the fact up, but finishing the sentence. That is why the most useful skill in the age of AI is not prompting, but checking.

Related on Poptani Labs: Why AI chatbots hallucinate — and why they can't just say 'I don't know', How AI image generators actually make pictures — and why the hands go wrong. More in this category: AI, Decoded.

Sources: Next-Token Prediction Explained: How LLMs Generate Text, How tokens work: ChatGPT never reads words, GPT-2: The Uncanny Valley of Text, How a Large Language Model Works: The Big Picture

← All posts