Semi-supervised GNN for Multi-modal Skin Lesion Classification

Abstract

Accurate and trustworthy classification of skin lesions is critical for early melanoma screening. While deep learning models have shown strong performance, they often underutilize clinical metadata, provide limited reliability cues, and face challenges under limited labeled data. This paper presents a trust-aware semi-supervised Graph Neural Network (GNN) framework for multi-modal skin lesion classification with three components: (1) a metadata-aware attention fusion mechanism integrating deep visual features, handcrafted descriptors, and clinical metadata; (2) entropy-based uncertainty estimation for trust calibration; and (3) a selective pseudo-labeling strategy that augments training using only high-trust unlabeled samples. Given the severe class imbalance of ISIC 2020, we emphasize clinically meaningful and class-balanced evaluation (e.g., Sensitivity, Specificity, Macro F1, and Macro AUC) rather than accuracy alone. Across repeated stratified cross-validation, we observe that multimodal metadata fusion does not necessarily maximize purely supervised Phase 1 performance, but provides more stable training dynamics and representations that support effective trust-aware refinement. In Phase 2, entropy-filtered pseudo-labeling yields consistent improvements in macro-level metrics for the selected backbone, indicating enhanced minority-class discrimination when high-trust samples are used. The proposed framework further supports reliability-oriented analysis by reporting trust-level performance and trust-conditioned confusion matrices, providing practical transparency on where predictions are more or less reliable. Limitations include incomplete clinical metadata and the use of entropy as a lightweight (but not fully optimal) uncertainty proxy under extreme imbalance; these aspects motivate future exploration of stronger calibration and transformer-based multimodal fusion.