Real time Semantic Segmentation for Human-Labeled Data: A comparative Study Between CNN and Transformer

Abstract

Convolutional Neural Networks have rapidly developed in the field of computer vision, notably in image classification, semantic segmentation, and object detection. These networks efficiently extract image features through local receptive fields and shared weights. Despite their effectiveness in various applications, CNNs face limitations, such as challenges in managing large-scale parameters and a tendency to overfit, especially in complex scenarios requiring contextual understanding. On the other hand, Transformer-based models, originally designed for natural language processing, have recently gained prominence in computer vision. They are particularly adept at capturing long-range dependencies, a critical aspect for interpreting complex visual scenes. Their scalability and adaptability open up new avenues for innovation. However, these models also come with drawbacks, including a need for extensive training data and higher computational costs. Their complex structures make optimization particularly challenging in resource-constrained environments. In this paper, our focus is on real-time model comparisons between CNN and Transformer architectures. We represent CNNs with the STDC model and Transformer-based models with the SegFormer. Our analysis revealed that the STDC model significantly outperforms in inference speed, achieving around 97 frames per second (fps), which is notably faster than the 50 fps of the SegFormer B0. However, when it comes to accuracy, the SegFormer B0 demonstrates superiority with a mean Intersection over Union (mIoU) of 86.78, which is more favorable compared to the 82.9 mIoU of the STDC. This study underscores the efficiency and accuracy trade-offs between these two architectures, highlighting the strengths of both CNNs and Transformer-based models in real-time applications.