MIRNet: Learning Multiple Identity Representations in Overlapped Speech

Many approaches exist for deriving a single speaker's identity information from speech by recognizing consistent characteristics of acoustic parameters. However, it is challenging to determine identity information when there are multiple concurrent speakers in a speech signal. In this paper, we propose a novel deep speaker representation strategy that can reliably extract multiple speaker identities from overlapped speech. We design a network that can extract a high-level embedding containing the identity information of each speaker from a given mixture. Unlike conventional approaches that need reference acoustic features for training, our proposed algorithm only requires the speaker identity labels for the overlapped speech segments. We demonstrate the effectiveness of our algorithm in a speaker verification task and a speech separation system conditioned on the target speaker embeddings obtained through the proposed method.