Universal Medical Image Segmentation using 3D Fabric Image Representation Encoding Networks

Data scarcity is a common issue for deep learning applied to medical image segmentation. One way to address this problem is to combine multiple datasets into a large training set and train a unified network that simultaneously learns from these datasets. This work proposes one such network, Fabric Image Representation Encoding Network (FIRENet), for simultaneous 3D multi-dataset segmentation. As medical image datasets can be extremely diverse in size and voxel spacing, FIRENet uses a 3D fabric latent module, which automatically encapsulates many multi-scale sub-architectures. An optimal combination of these sub-architectures is implicitly learnt to enhance the performance across many datasets. To further promote diverse-scale 3D feature extraction, a 3D extension of atrous spatial pyramid pooling is used within each fabric node to provide a finer coverage of rich-scale image features. In this study, FIRENet was first applied to 3D universal bone segmentation involving multiple musculoskeletal datasets of the human knee, shoulder and hip joints. FIRENet exhibited excellent universal bone segmentation performance across all the different joint datasets. When transfer learning is used, FIRENet exhibited both excellent single dataset performance during pre-training (on a prostate dataset) as well as significantly improved universal bone segmentation performance. In a following experiment which involves the simultaneous segmentation of the 10 Medical Segmentation Decathlon (MSD) challenge datasets. FIRENet produced good multi-dataset segmentation results and demonstrated excellent inter-dataset adaptability despite highly diverse image sizes and features. Across these experiments, FIRENet's versatile design streamlined multi-dataset segmentation into one unified network. Whereas traditionally, similar tasks would often require multiple separately trained networks.