A research paper, entitled “MedIM: Boost Medical Image Representation via Radiology Report-guided Masking” was accepted in top medical AI conference, The 26th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2023.
Masked image modelling (MIM)-based pre-training shows promise in improving image representations with limited annotated data by randomly masking image patches and reconstructing them. However, random masking may not be suitable for medical images due to their unique pathology characteristics. This paper proposes the first approach that masks and reconstructs discriminative areas guided by radiological reports, encouraging the network to explore the stronger semantic representations from medical images. The extensive experiments on various downstream tasks covering multi-label/class image classification, medical image segmentation, and medical image-text analysis, demonstrate that proposed method with report-guided masking achieves competitive performance. This method substantially outperforms ImageNet pre-training, MIM-based pre-training, and medical image-report pre-training counterparts.
- Authors:
Yutong Xie ( University of Adelaide )
Lin Gu ( RIKEN AIP)
Tatsuya Harada ( The University of Tokyo / RIKEN AIP)
Jianpeng Zhang ( Alibaba DAMO Academy )
Yong Xia ( Northwestern Polytechnical University )
Qi Wu ( University of Adelaide )
