Self-supervised Co-learning of Uncurated Images and Reports Enables Oversight AI in Radiology

Oversight AI is an emerging concept in radiology where the AI forms a symbiosis with radiologists by continuously supporting radiologists in their decision-making. Recent advances in vision-language pre-training sheds a light on the long-standing problems of the oversight AI by the understanding of both visual and textual concepts and their semantic correspondences. However, there have been limited successes in the application of vision-language pre-training in the medical domain, as the current vision-language models and learning strategies for photographic images and captions are not optimal to process the medical data that are usually insufficient in the amount and the diversity. To address this, here we present medical X-VL, a self-supervised model tailored for efficient vision-language pre-training that exploits cross attention in the radiological images and reports' common feature space in a symmetric manner. We experimentally demonstrate that the pre-trained medical X-VL model outperforms the current state-of-the-art models in various vision-language tasks in medical domains. We finally demonstrate practical clinical usages of our oversight AI for monitoring human errors and in the diagnosis of newly emerging diseases, which suggests the potential of an oversight AI model for widespread applicability in different medical applications.