Neural Representations for Modeling Variation in Speech

Variation in speech is often represented and investigated using phonetic transcriptions, but transcribing speech is time-consuming and error prone. As an alternative representation, therefore, we investigate the extraction of acoustic embeddings from several self-supervised neural models. We use these representations to compute word-based pronunciation differences between non-native and native speakers of English, and between different dialect pronunciations, and evaluate these differences by comparing them with available human native-likeness judgments. We show that Transformer-based speech representations lead to significant performance gains over the use of phonetic transcriptions, and find that feature-based use of Transformer models is most effective with one of the middle layers instead of the final layer. We also demonstrate that these neural speech representations not only capture segmental differences, but also intonational and durational differences that cannot be represented by a set of discrete symbols used in phonetic transcriptions.