Blog

• May 25, 2026
  What RMSNorm Actually Does (And Why You Can't Replace It With Tanh)

  An empirical dissection of normalization in transformers, and why the backward coupling, not the forward rescaling, is what makes RMSNorm work.