What is an LLM's architecture?

An LLM's architecture is the underlying design of the neural network that makes it work — and for virtually every modern LLM, that design is the transformer. Introduced by Google researchers in 2017, the transformer is the breakthrough that made today's language models possible, and it's what the T in GPT stands for.

What makes the transformer special?

The key is a mechanism called attention. Instead of reading text strictly left to right, attention lets the model weigh how much every word should influence every other word, all at once. That's how it keeps track of context across long passages — knowing that it refers to a noun mentioned three sentences ago, for example.

Earlier designs read word by word and tended to forget the start of a long passage by the time they reached the end. Attention removed that bottleneck, which is why the transformer replaced the approaches that came before it.

How is a transformer built?

A transformer is built from many identical layers stacked on top of each other. Each layer refines the model's understanding of the input a little more, passing richer representations upward:

• Lower layers tend to capture basic patterns like grammar and word order.
• Higher layers build toward meaning, references, and the gist of a passage.

Stack enough layers and feed in enough data, and the model develops a startlingly deep grasp of language.

Why does this design scale so well?

The other key trait is parallelism: transformers process an entire input at once rather than word by word, which makes them efficient to train on modern hardware built for doing many calculations in parallel. That efficiency is a big reason models could be scaled up to the enormous sizes that gave rise to their surprising abilities. The same architecture works whether the model is small or huge — you mostly add more layers, more data, and more compute — which is why one design has come to define nearly the entire field.