Pre-trained Language Models in Biomedical Domain: A Survey from Multiscale Perspective

Pre-trained language models have been the de facto paradigm for most natural language processing (NLP) tasks. In the biomedical domain, which also benefits from NLP techniques, various pre-trained language models were proposed by leveraging domain datasets including biomedical literature, biomedical social medial, electronic health records, and other biological sequences. Large amounts of efforts have been explored on applying these biomedical pre-trained language models to downstream biomedical tasks, from informatics, medicine, and computer science (CS) communities. However, it seems that the vast majority of existing works are isolated from each other probably because of the cross-discipline characteristics. It is expected to propose a survey that not only systematically reviews recent advances of biomedical pre-trained language models and their applications but also standardizes terminology, taxonomy, and benchmarks. Therefore, this paper summarizes the recent progress of pre-trained language models used in the biomedical domain. Particularly, an overview and taxonomy of existing biomedical pre-trained language models as well as their applications in biomedical downstream tasks are exhaustively discussed. At last, we illustrate various limitations and future trends, which we hope can provide inspiration for the future research.