$ why deep --honest
one line → fails on XOR. proven on /ml/.
hidden layers → fold space until the line works
activation → the bend has a name, and a dead zone
attention → tokens scoring tokens. a lookup, not magic
$ rule
depth is not depth of thought. it is depth of folding.
$ live
status: gradients flowing (mostly)DL / visual intuition
fold space until the rule works.
The ML page showed where one line dies. This page shows the machinery that replaces it — every trick made visible, no fog.
the story so far
from a failing line to a mapped fly brain.
I didn’t read this history — I rebuilt each step so the failure and the fix are visible. That’s the point.
chapter 00 / the fix
XOR, solved live in front of you.
Same cursed dataset as the homepage — but now a 2-2-1 network trains with real backprop in your browser. Watch the boundary bend itself into existence.
Training a real 2→2→1 network on XOR with backpropagation. The white contour is where the network is exactly unsure — watch it fold.
h = tanh(W₁x + b₁) ŷ = σ(W₂h + b₂)
Every frame is one real gradient step. Nothing is scripted — if it solves XOR, it solved XOR.
chapter 01 / the bend
an activation is a bend with a dead zone.
Sigmoid squishes. Tanh squishes symmetrically. ReLU just chops. The dashed curve is the gradient — where it flatlines, learning dies. That is why everyone stopped using sigmoid in hidden layers.
σ(z) = 1/(1+e⁻ᶻ) → σ′ maxes at 0.25
relu(z) = max(0,z) → gradient is 0 or 1
- max gradient
- —
- dead zone
- —
chapter 02 / convolution
a CNN is a stencil that learned what to look for.
A 3×3 kernel slides over the image; each stop produces one number in the feature map. Same weights everywhere — that weight sharing is the entire trick. Watch an edge detector do its rounds.
out[i,j] = Σ kernel ⊙ patch(i,j)
shared weights → translation invariance
chapter 03 / attention
attention is a lookup, not magic.
Every token scores every token: Q·K, scaled, softmaxed. Hover a cell to read the score. The bright rows show who each word listens to.
softmax(Q·Kᵀ / √d) · V
Each cell is how much the row-token listens to the column-token. Raw score first, softmax after — the difference between opinion and attention.
chapter 04 / the terrain
same loss, two learning rates, two fates.
Gradient descent on a real loss surface with a valley and a ridge. The cautious rate creeps down the ravine. The ambitious rate overshoots, ricochets, and sometimes escapes to a worse basin. Step size is a personality trait.
w ← w − η·∇L
η small: slow but loyal / η big: fast but chaotic
- careful η end loss
- —
- bold η end loss
- —
chapter 05 / real digit recognizer
draw a digit at a trained model.
A linear softmax readout — 1024 pixels → 10 scores — trained live in this tab on 130 augmented font-rendered digits. Not a demo GIF: the weights are learned while you watch, then they judge your drawing.
- final train loss
- —
- synthetic test acc
- —
- status
- queued
chapter 06 / a real brain
this is not a simulation. it’s a fly.
FlyWire’s whole-brain connectome (Nature 2024): 139,255 proofread neurons, 50M+ synapses. Below is the real wiring aggregated to 28 brain regions — hover any cell. This network was never trained. It grew.
Virtual Fly Brain mesh + surface pulse points. Move the pointer over it — this is the brain shape, not the matrix.
Each cell = synapses from region (row) to region (column), log-scaled brightness. The diagonal glows because regions mostly talk to themselves — the same small-world trick your MLP accidentally reinvented.
48.9M synapses mapped · 28 regions
Source: FlyWire Consortium, proofread_connections_783 (Zenodo 10676866). gaba / ach / glut = dominant neurotransmitter share per edge.
chapter 08 / connectome explorer 2.0
trace the fly brain like a network, not a wallpaper.
The FlyWire matrix becomes an actual graph lab: isolate regions, follow strongest synapse paths, rank central hubs, then compare the grown brain to an artificial layered net.
Building weighted region graph from the same FlyWire aggregate as the matrix above.
exit
depth folds. evolution folds harder.
You saw a line die, a bend fail, a stencil see, tokens listen, a step size choose a fate, a readout layer train itself in your tab — and then a brain that was never trained at all, just grown into the same small-world shape. That’s the whole trick.