Blog
- What Actually Changes When a Network GroksGrokking on a new non-abelian group task, and an anatomy of the transition: the collapse of a memorising lookup table onto about three irreducible representations, gated by weight decay, timed by the learning rate, and preceded by no warning at all.
- What RMSNorm Actually Does (And Why You Can't Replace It With a Point-wise Norm)An empirical dissection of normalization in transformers, and why the backward coupling, not the forward rescaling, is what makes RMSNorm work.