In this paper, we are interested in unsupervised (unknown noise) audio-visual speech enhancement based on variational autoencoders (VAEs), where the probability distribution of clean speech spectra is simulated using an encoder-decoder architecture. The trained generative model (decoder) is then combined with a noise model at test time to estimate the clean speech. In the speech enhancement phase (test time), the initialization of the latent variables, which describe the generative process of clean speech via decoder, is crucial, as the overall inference problem is non-convex. This is usually done by using the output of the trained encoder where the noisy audio and clean visual data are given as input. Current audio-visual VAE models do not provide an effective initialization because the two modalities are tightly coupled (concatenated) in the associated architectures. To overcome this issue, inspired by mixture models, we introduce the mixture of inference networks variational autoencoder (MIN-VAE). Two encoder networks input, respectively, audio and visual data, and the posterior of the latent variables is modeled as a mixture of two Gaussian distributions output from each encoder network. The mixture variable is also latent, and therefore the inference of learning the optimal balance between the audio and visual inference networks is unsupervised as well. By training a shared decoder, the overall network learns to adaptively fuse the two modalities. Moreover, at test time, the visual encoder, which takes (clean) visual data, is used for initialization. A variational inference approach is derived to train the proposed generative model. Thanks to the novel inference procedure and the robust initialization, the proposed MIN-VAE exhibits superior performance on speech enhancement than using the standard audio-only as well as audio-visual counterparts.
翻译:在本文中,我们感兴趣的是基于变异自动读取器(VAEs)的未经监督(未知噪音)的视听语音增强,其基础是变异自动读取器(VAEs),清洁语音光谱的概率分布通常是通过一个编码解码器结构模拟的。经过培训的基因化模型(Decoder)随后与测试时的噪音模型结合,以估计干净的言语。在语音增强阶段(测试时间),潜在变量的初始化至关重要,它描述了通过解码器进行清洁语音变异过程的基因化过程,因为总体推断问题是非convex。这通常是通过使用经过培训的视觉解析器的编码模式的输出来完成的。当前视听VAEE模型没有提供有效的初始化模型,在相关结构中,两种声音变异变码网络的高级变异性(Order)是模拟变现的变现模型,因此,变现的变现的变现变现数据是预变现模型中的变现变现。