From a single neuron to DiT: how image models learned to see, then to draw.

Every image model you have heard of, from the toy that recognises a handwritten 7 to the one that paints a photorealistic astronaut on a horse, is built from the same handful of ideas stacked in ever more clever ways. This is a walk up that staircase, one step at a time. We start with a single neuron, add layers, teach it to see with convolutions, let it go deep without falling over, then flip the whole thing around so it learns to draw instead of to label. By the end we arrive at the diffusion transformer, and it will not feel like magic. It will feel like the obvious next step.

The thread running through all of it is one question that keeps getting reframed: what shape should the computation take so that gradients can flow, information can travel, and structure can be learned? Each architecture in this post is an answer to a specific failure of the one before it. Hold that framing and the whole history clicks into place.

01The single neuron

Everything begins with one neuron, and a neuron is embarrassingly simple. It takes some numbers in, multiplies each by a weight, adds them up, adds one more number called a bias, and passes the total through a squashing function. That is the entire object.

The weights w say how much each input matters. The bias b shifts the threshold at which the neuron gets excited. The activation function σ is what makes it more than a spreadsheet formula: it bends the straight line into a curve, so the neuron can express something other than pure proportion. Without it, no matter how many neurons you stack, the whole network collapses back into a single linear equation. The non-linearity is the thing that lets depth mean something.

A single neuron with a sigmoid is logistic regression, and it can separate data with a single straight line. That is genuinely useful, and also a hard ceiling. Draw two classes that curl around each other, the classic XOR pattern, and one straight line can never split them. The fix is not a cleverer neuron. It is more neurons, arranged in layers.

02Stacking neurons: the MLP and MNIST

Put many neurons side by side and you get a layer. Stack layers and you get a multi-layer perceptron, the MLP. Each neuron in a layer looks at every output from the layer before it, which is why these are called fully connected or dense layers. The first hidden layer might learn to detect little strokes and curves; the next combines strokes into shapes; the last combines shapes into a decision.

The canonical first problem is MNIST: 70,000 tiny 28×28 greyscale images of handwritten digits, and the job is to name each one from 0 to 9. You flatten the 28×28 grid into a flat vector of 784 numbers, feed it through a couple of hidden layers, and end with 10 output neurons, one per digit. A softmax on those ten turns raw scores into probabilities that sum to one, and you pick the largest.

Two hidden layers of this kind will hit around 98% on MNIST, which feels like a triumph until you notice what the flattening step quietly destroyed. The instant you unroll a 28×28 grid into 784 numbers in a row, you tell the network that the pixel at position 30 and the pixel directly below it (position 58) are just two entries in a list, no more related than positions 1 and 700. All spatial structure is gone. The network has to relearn, from scratch and separately for every location, that nearby pixels tend to belong together.

03Teaching it to see: the CNN and cat detection

The convolutional neural network fixes both problems with one idea: instead of giving every pixel its own private weights, slide a small window of shared weights across the whole image. That little window, the kernel or filter, is just a tiny grid of numbers, maybe 3×3. You place it over one patch, multiply and sum (that is one convolution), then slide it one step over and do it again. The same kernel sweeps the entire image.

Two enormous wins fall out of this. First, weight sharing: a kernel that learns to detect a vertical edge detects that edge everywhere, because the same nine numbers are reused at every position. A cat in the corner and a cat in the centre light up the same filters. Second, locality: each output looks only at a small neighbourhood, which is exactly how visual structure works. Edges make textures, textures make parts, parts make objects.

You stack these. Early convolution layers learn edges and blobs. Pooling layers throw away spatial resolution to keep the important activations and grant a little tolerance to shifts. As you go deeper the feature maps get spatially smaller but semantically richer, and their depth (the number of distinct feature channels) grows. Early layers see edges; middle layers see fur texture and ear shapes; late layers fire on whiskers, pointy ears, cat face. Flatten that final rich stack, run it through a small dense head, and out comes cat or not-cat.

This template, convolution and pooling into a dense head, is LeNet from 1998. Scaled up, trained on ImageNet's million photos, and run on GPUs, it became AlexNet in 2012 and cut the image-recognition error rate almost in half overnight. That single result is what kicked off the modern era. Then people tried the obvious thing: if deep is good, deeper must be better. And it broke.

04Going deeper without falling over: ResNet

Stacking more layers should never make a network worse. In the limit, the extra layers could just copy their input forward and match the shallower model. Yet in practice, a 56-layer plain network trained worse than a 20-layer one, on both training and test data. The problem was not overfitting. It was that gradients, the signals that tell each layer how to adjust, had to travel back through dozens of multiplications to reach the early layers, and along the way they shrank toward zero. The vanishing gradient. Deep networks were untrainable not because they lacked capacity but because the learning signal could not reach the bottom.

ResNet's fix is one of the most quietly influential ideas in the whole field, and it is almost trivially simple. Add a shortcut that skips over a couple of layers and adds the input straight to their output.

Now the layers only have to learn the residual, the small change F(x) to make to the input, rather than the entire useful transformation from scratch. If the best thing a block can do is nothing, it just learns F(x) = 0 and the identity passes through untouched. And crucially, that + x gives the gradient a clean highway straight back to earlier layers, bypassing the multiplications that were killing it.

With residual connections, networks jumped from around 20 usable layers to 152, then to a thousand in research settings, and accuracy kept climbing. The skip connection turned out to matter far beyond classification. Hold onto it, because the same idea reappears twice more in this post, first as the crossbars of the U-Net, then inside every transformer block.

05Flipping the arrow: from recognising to generating

Every network up to here is discriminative: image in, label out. Generation is the reverse, and it is a genuinely harder problem. There is exactly one correct answer to is this a cat? There are billions of correct answers to draw me a cat. The model has to learn not a boundary between classes but the entire landscape of what plausible images look like, and then sample new points from it.

The first bridge is the autoencoder. Take an image, squeeze it through a narrow bottleneck with an encoder, then reconstruct it with a decoder, and train the output to match the input. The network is forced to compress the image into a compact code and rebuild it, so that small code has to capture the essence. The variational autoencoder (VAE) adds a probabilistic twist: it makes that code a smooth, well-behaved region rather than a scattering of isolated points, so you can sample a random code and decode it into a brand-new image.

Around the same time, GANs pitted two networks against each other, a generator inventing fakes and a discriminator trying to catch them, and produced sharp results but were famously unstable to train. The autoencoder idea is what carries forward, for two reasons that matter enormously later. First, the decoder shows you can go from a compact code back to a full image. Second, and this is the key architectural donation, the encoder-decoder shape with a bottleneck became the U-Net.

The U-Net: an encoder-decoder with a memory

The U-Net was invented for medical image segmentation, where you need a pixel-perfect output the same size as the input. Its shape is an encoder that downsamples the image into a small, deep, semantic representation, and a decoder that upsamples back to full resolution. The problem: the downsampling path throws away fine spatial detail (exact edges, precise positions) in exchange for meaning. The decoder, rebuilding from the tiny bottleneck, would produce a blurry mess.

The fix is the same trick as ResNet, applied across the whole network. At each resolution, run a skip connection straight across from the encoder to the matching decoder level, handing the fine detail directly over. The decoder gets the best of both: high-level meaning from the bottleneck, sharp detail from the skips. That is the U shape, and it is the workhorse of the first generation of diffusion models.

06The idea that changed everything: attention

Before we assemble a real image generator, we need the one building block that the advanced models are organised around. It came from language, not vision. The problem in translation was that a word's meaning depends on other words, sometimes far away in the sentence. Convolutions only look locally; recurrent networks carried information one step at a time and forgot. Attention let every element look directly at every other element and decide, per pair, how much to care.

The mechanism is built from three projections of the input, with names borrowed from databases. For each element you compute a query (what am I looking for?), a key (what do I offer?), and a value (what will I actually hand over?). Every query is compared against every key by a dot product, giving a grid of match scores. Softmax turns each row of scores into weights that sum to one, and the output for each element is the weighted blend of all the values.

Read it left to right. Q Kᵀ is every-query-against-every-key, the full grid of affinities. Dividing by √d keeps the numbers in a sane range so the softmax does not saturate. Softmax normalises each row into attention weights. Multiplying by V mixes the values according to those weights. In one shot, every element has gathered information from every other element, near or far.

Two variants matter for us. When Q, K and V all come from the same sequence, it is self-attention: the image talking to itself, letting distant regions coordinate. When Q comes from one source and K, V come from another, it is cross-attention: the exact mechanism by which a text prompt reaches in and steers an image. That single distinction is what turns a denoiser into a text-to-image model. In 2017 the transformer paper showed you could throw away convolutions and recurrence entirely and build a whole model out of attention plus small dense layers. Vision would follow.

07Putting it together: Stable Diffusion

Now we have every piece. Diffusion is the training idea that finally made image generation stable and controllable, and its recipe is almost suspiciously simple. Take a real image and add a little Gaussian noise. Add a little more. Keep going for many steps until nothing is left but static. That destruction process is fixed and requires no learning. Then train a network to do one thing: given a noisy image and a timestep, predict the noise that was added. Subtract the predicted noise and you have stepped one notch back toward a clean image.

To generate, start from pure random static and run that trained denoiser over and over, each pass removing a bit more noise, until a coherent image emerges. It is sculpture: start with a formless block and repeatedly chip away what is not the statue. (For the intuition on why a humble Gaussian is exactly the right tool here, I wrote a whole separate piece, Why a Gaussian Can't Model Faces, But Can Model Diffusion.)

Stable Diffusion added the three refinements that made this practical on a normal GPU, and each one is a building block we have already met.

• Do it in latent space, not pixel space (the VAE). Denoising a 512×512×3 image directly is enormously expensive. So Stable Diffusion first uses a VAE encoder to compress the image into a small latent grid, roughly 64×64, runs the entire slow diffusion process there, then decodes back to full resolution once at the end. This is the single change that put image generation within reach of consumer hardware, hence the latent diffusion name.
• Make the denoiser a U-Net. The network that predicts the noise is exactly the encoder-decoder-with-skips from earlier, operating on the latent grid. Its job is pixel-shaped (noise in, noise out, same size), which is precisely what the U-Net was born to do.
• Steer it with text via cross-attention (CLIP). The prompt is encoded by a CLIP text encoder into a sequence of embeddings. Cross-attention layers are inserted throughout the U-Net, where the image latent forms the queries and the text embeddings form the keys and values. At every denoising step, every region of the image asks the prompt what should I be becoming? and pulls the answer in.

Look at what that diagram is made of. A VAE (autoencoder, step 5). A U-Net (encoder-decoder with skips, step 5) whose blocks are stuffed with residual connections (step 4), convolutions (step 3), self-attention and cross-attention (step 6). Nothing here is new. Stable Diffusion is an assembly of the ideas on the staircase, arranged so that noise plus a prompt becomes a picture.

08Scaling it up: SDXL

SDXL is not a new idea, it is the same architecture built bigger and smarter, and it is a good lesson in how much raw scale plus a few targeted fixes can buy you. The backbone is still a latent-diffusion U-Net with cross-attention. What changed is where the capacity went and how the conditioning got richer.

The cleverest additions are the small conditioning tricks. Earlier models trained by cropping and resizing images, which quietly taught them bad habits (cut-off heads from random crops, a bias toward low resolution from discarding small training images). SDXL fixes this with micro-conditioning: it feeds the original image size and the crop coordinates to the model as extra conditioning signals, alongside the timestep. Now the model knows this training example was a top-left crop of a small image and can learn to associate artifacts with those signals, so at generation time you simply ask for a centred, full-size, high-resolution image and it obliges. Finally, an optional second refiner model, a small specialist diffusion model, polishes the last few denoising steps for fine detail.

SDXL is the high-water mark of the U-Net era. But notice a tension building. The most powerful parts of the network are the attention blocks, and the whole rest of the U-Net (the convolutions, the hand-designed multi-resolution scaffolding, the skip crossbars) is essentially plumbing built around them. That raises an obvious question. If attention is doing the heavy lifting, what happens if we throw the U-Net away entirely?

09The clean rewrite: the Diffusion Transformer (DiT)

The Diffusion Transformer answers that question directly: keep the diffusion recipe and the latent space, but replace the U-Net denoiser with a plain transformer, the same kind of architecture that was already dominating language. The convolutional scaffolding is gone. What is left is almost entirely attention.

The move that makes this possible is the same one Vision Transformers used for classification: patchify. Cut the latent grid into a checkerboard of small patches, flatten each patch into a vector, add a positional embedding so the model knows where each patch sat, and you now have a sequence of tokens, exactly the diet a transformer eats. From there it is a stack of identical transformer blocks: self-attention lets every patch talk to every other patch (global context from layer one, something the U-Net only achieved at its bottleneck), followed by a small dense network, each wrapped in residual connections.

The one genuinely new piece is how you inject the conditioning, the timestep and the class or text. A U-Net splices in cross-attention layers. DiT's most effective variant, adaLN-Zero, does something slicker. It feeds the conditioning through a small network that outputs a set of scale, shift, and gate numbers, and uses them to modulate the normalisation inside each block (this is adaptive layer norm). The Zero part is a lovely detail: the gates are initialised so that every block starts as an identity function, doing nothing, and the network gently learns how much of each block to switch on. It is the residual-block philosophy, "start from identity, learn the change", reincarnated as a conditioning scheme.

Why did this matter beyond elegance? Because transformers scale in a way the U-Net never cleanly did. The DiT paper showed a smooth, predictable relationship: pour in more compute, more parameters, more attention, and image quality improves reliably, no architectural surgery required. That is the property that language models had been riding for years, now available for images. It is the reason the newest generation of image and video models (including the architecture family behind Stable Diffusion 3 and Sora) are transformer-based rather than U-Net-based. The clean rewrite won.

10The whole staircase, in one breath

Step back and the evolution reads as a single sentence getting longer. A neuron is a weighted sum through a curve. Stack them into an MLP and you can classify, until images defeat you. Share the weights and slide them, and the CNN learns to see. Add a skip connection and ResNet lets you go arbitrarily deep. Bend the encoder-decoder into a U-Net and you can output a full image, not just a label. Bolt on attention and every part of the image can consult every other part, and a text prompt can consult all of them. Wrap that in the diffusion recipe and run it in a compressed latent space, and you have Stable Diffusion. Scale it and you have SDXL. Throw away the convolutional scaffolding, keep only the attention, and you have DiT.

Not one of those steps is a magic leap. Each is a specific, legible fix for the failure of the step before it. The models that paint dreamlike images today are the same neuron from 1958, multiplied a few hundred billion times and wired with a handful of very good ideas about how information should flow. That is the whole trick, and once you can see the staircase, none of the steps look like magic anymore.

From one weighted sum to a transformer that dreams in pixels. Same idea, better plumbing, all the way up.Thanks for reading ✦

Architectures referenced, roughly in order: the perceptron (Rosenblatt, 1958), backpropagation and MLPs (Rumelhart et al., 1986), LeNet (LeCun et al., 1998), AlexNet (Krizhevsky et al., 2012), ResNet (He et al., 2015), U-Net (Ronneberger et al., 2015), the transformer and attention (Vaswani et al., 2017), the Vision Transformer (Dosovitskiy et al., 2020), DDPM diffusion (Ho et al., 2020), latent / Stable Diffusion (Rombach et al., 2022), SDXL (Podell et al., 2023), and DiT (Peebles & Xie, 2023). Diagrams are schematic and omit many details for clarity.