SDPA is Entropy-Regularized Optimal Transport
Softmax attention is the unique optimal solution to an entropy-regularized transport problem. This connects 18th-century French mathematics, reinforcement learning, and the geometry of probability distributions.
attentionoptimal-transportinformation-geometrytransformers