*in machine learning
\nElijah Potter | October 16, 2026
\nOne of my favorite blog posts of all time was written by Andrej Karpathy nearly ten years ago.\nIn it, he details how he would require the students of his CS231n Deep Learning class at Stanford to implement the forward and backward passes of neural networks by hand (that is, in Python with NumPy).\nWhy?\nBecause he wanted his students to leave his class with a visceral understanding of backpropagation.\nThey should know how it works, why it works, and thus how to fix it when it stops working.
\nIt’s a great read, and I highly recommend that anyone and everyone interested in ML peruse it.\nYou can find it here.
\nThere are some things that are sealed abstractions (that is, they are not leaky).\nBut I do not think any of them exist in the serious study of machine learning.\nSome examples include:
\nSure, with some of these things you occasionally need to understand the underlying system for performance tuning.\nBut knowledge of the underlying system is rarely the difference between success and failure.
\nML is not that way.\nTo successfully train a model, you need to have a clear understanding of every aspect of the system, all the way down to how individual weights and biases are updated.\nIt is not a kind discipline.
\nObviously, there is Karpathy’s example of backpropagation, but it goes further than that.\nEvery aspect of model architecture can influence the shape of the cost function optimized via backpropagation, which means every aspect of model architecture is leaky.
\nI have got the sense that laymen believe that any problem can be solved by throwing a transformer at it.\nIf only it were so simple.
\nSuccessful use of a transformer (or any model architecture, for that matter) depends on a good understanding of how it works and how it affects the cost function.
\nAs with any kind of system with a Rube Goldberg–level complexity, it is best to take your time.\nGo slow.\nSkip that extra pot of coffee.\nIt’s better to be right slowly than wrong very fast.\nUnderstand every aspect of your model and training regimen before you start, not after.
\nPublished October 16, 2026 at 9:08 PM
\nProofread by Harper.
","headings":[{"level":1,"text":"Everything Is a Leaky Abstraction*","id":"everything-is-a-leaky-abstraction"},{"level":2,"text":"Some Sealed Abstractions","id":"some-sealed-abstractions"},{"level":2,"text":"ML Is Not like Normal Engineering","id":"ml-is-not-like-normal-engineering"},{"level":2,"text":"Take Your Time","id":"take-your-time"}]}}