Path-Constrained Mixture-of-Experts proposes a new lens for understanding MoE architectures. Researchers observed that tokens in practice utilize a surprisingly small subset of possible expert paths, revealing statistical inefficiencies. This work motivates new architectures designed to better constrain the effective path space for improved efficiency.
Opening Kapyn…