I Built a Deep Learning Framework From Scratch. Now I Read Every Model Differently.

For most of my degree, PyTorch was a place where gradients came from. You call .backward(), the numbers appear, the model learns. It works, so you stop asking how.

Then I took the deep learning course at Berkeley where the whole point was the opposite. Not use the framework. Build it. Over a semester I wrote a small deep learning library from numpy up: the tensor, the autograd engine, the layers, the optimizers. By the end, .backward() was not magic. It was code I had written and debugged at 2am.

The library is needle, a numpy-based framework I built from scratch: reverse-mode automatic differentiation, twenty operators, and the usual layers and optimizers. Trained on gradients my own engine computed, it lands within half a percent of PyTorch’s accuracy on MNIST, at roughly seventeen times the runtime, since pure numpy has no BLAS to lean on. The first time that worked, something reorganized in my head.

The foundation was the point

Here is what I did not expect. The class was not really about deep learning. It was about the foundation, and the foundation changed how I read everything after it.

Once you have built reverse-mode autodiff by hand, you cannot look at a training loop the same way. You know what a computation graph is because you built the nodes. You know why a gradient explodes because you watched the chain rule multiply the wrong numbers into each other. You know what an optimizer actually touches when it steps. The abstractions stop being walls and start having an inside.

Using a tool and understanding it are different jobs, and the second one compounds. Every paper I read now, I read for the mechanism. Every model I debug, I have a picture of what is actually moving. The Transformer I built for translation right after, an encoder-decoder that scores around 28 BLEU on English to German, was hard, but it was hard in a way I could reason about, because I already knew what attention was doing to the tensors underneath.

Why I keep coming back to it

I write a lot here about how the model is the small part, how the harness and the plumbing and the reliability are the real engineering. That instinct did not come from nowhere. It started in this class, where I learned that the impressive black box is just code someone wrote, and that if you rebuild it once, it stops being a black box for good. The same split showed up later when I wired two notebook agents and when a provably optimal scheduler I wrote still could not ship at the size that mattered.

The most useful thing a class can do is rebuild your intuition from the floor, so everything you learn next lands somewhere solid. This one did that for me, and I have been building on it ever since. Both projects are on my GitHub if you want to see what came out of it.

I have enjoyed few things in school as much as the week I finally got backprop working by hand.