New research proposes methods to improve attention-based models for long audio inputs. The study identifies a fundamental incompatibility with long-form acoustic encodings, causing models to lose ordering capabilities due to permutation invariance. Four modifications, including explicit absolute positional encodings, are introduced to address this limitation.
Opening Kapyn…