Blog

• May 25, 2026
  What RMSNorm Actually Does (And Why You Can't Replace It With a Point-wise Norm)

  An empirical dissection of normalization in transformers, and why the backward coupling, not the forward rescaling, is what makes RMSNorm work.