A novel attention mechanism for noise-adaptive and robust segmentation of microtubules in microscopy images

Segmenting cytoskeletal filaments in microscopy images is essential for understanding their cellular roles but remains challenging, especially in dense, complex networks and under noisy or low-contrast image conditions. While deep learning has advanced image segmentation, performance often degrades in these adverse scenarios. Additional challenges include the difficulty of obtaining accurate annotations and managing severe class imbalance. We proposed a novel noise-adaptive attention mechanism, extending the Squeeze-and-Excitation (SE) module, to dynamically adjust to varying noise levels. This Adaptive SE (ASE) mechanism is integrated into a U-Net decoder, with residual encoder blocks, forming a lightweight yet powerful model: ASE_Res_U-Net. We also developed a synthetic-dataset strategy and employed tailored loss functions and evaluation metrics to mitigate class imbalance and ensure fair assessment. ASE_Res_U-Net effectively segmented microtubules in both synthetic and real noisy images, outperforming its ablated variants and state-of-the-art curvilinear-structure segmentation methods. It achieved this while using fewer parameters, making it suitable for resource-constrained environments. Importantly, ASE_Res_U-Net generalised well to other curvilinear structures (blood vessels and nerves) under diverse imaging conditions. Availability and implementation: Original microtubule datasets (synthetic and real noisy images) are available on Zenodo (DOIs: 10.5281/zenodo.14696279 and 10.5281/zenodo.15852660). ASE_Res_UNet model will be shared upon publication.