We propose a learnable content adaptive front end for audio signal processing. Before the modern advent of deep learning, we used fixed representation non-learnable front-ends like spectrogram or mel-spectrogram with/without neural architectures. With convolutional architectures supporting various applications such as ASR and acoustic scene understanding, a shift to a learnable front ends occurred in which both the type of basis functions and the weight were learned from scratch and optimized for the particular task of interest. With the shift to transformer-based architectures with no convolutional blocks present, a linear layer projects small waveform patches onto a small latent dimension before feeding them to a transformer architecture. In this work, we propose a way of computing a content-adaptive learnable time-frequency representation. We pass each audio signal through a bank of convolutional filters, each giving a fixed-dimensional vector. It is akin to learning a bank of finite impulse-response filterbanks and passing the input signal through the optimum filter bank depending on the content of the input signal. A content-adaptive learnable time-frequency representation may be more broadly applicable, beyond the experiments in this paper.
翻译:我们提出了一种学习型内容自适应前端,用于音频信号处理。在深度学习盛行之前,我们使用了不可学习的固定表示法前端,如谱图或带/不带神经架构的梅尔谱图。卷积神经网络支持各种应用,如ASR和声场理解,它们开始采用可学习的前端,既从头开始学习基函数类型,又从头开始学习权重,并针对特定的任务进行优化。随着基于转换器的架构的到来,不再存在卷积块,线性层将小的波形补丁投影到小的潜在维度中,然后将它们送入转换器架构。在这项工作中,我们提出了一种计算内容自适应的可学习时频表示法的方法。我们将每个音频信号通过一组卷积滤波器,每个滤波器都给出一个固定维度的向量。这类似于学习一组有限脉冲响应滤波器组并根据输入信号的内容将其通过最佳滤波器组。内容自适应的学习型时频表示法可能具有更广泛的适用性,超越了本文的实验。