Prompt Driven Multimodal Large Language Models for Concrete Defect Identification

Abstract

Detecting surface defects in concrete structures is essential for ensuring structural safety; however, traditional methods have limitations, including subjectivity and heavy data dependency. This study proposes using pretrained multimodal large and small language models (LLM/SLM), specifically GPT-4o and Gemini 2.5-Flash, for concrete defect detection without additional fine-tuning. Structured prompts (S-Prompt) were employed to direct the task instructions and to elicit structured JSON outputs. The models’ performances were evaluated in both zero-shot and few-shot scenarios, the latter using a compact exemplar board. Performance was benchmarked against a fine-tuned YOLOv8 model, employing standard detection metrics alongside newly introduced metrics: Class Presence Accuracy (CPA), Relaxed Localization Recall (RLR), and Unmatched Prediction Ratio (UPR). Experimental results demonstrated that even in a zero-shot setting, the LLM/SLM models meaningfully identified defect presence, type, and approximate location (at an IoU threshold of 0.1) (GPT-4o: F1@ 0.1=0.468 , CPA-F 1=0.667 ). The few-shot scenario improved certain metrics (F1@0.1, CPA-F1, RLR@0.1, etc.) with reduced UPR@0.1, but exhibited a significant trade-off in reduced inference speed. Although the proposed method showed lower performance than YOLOv8 in precise localization tasks (higher IoU thresholds), it demonstrated significant potential as a training-free pipeline suitable for initial screening purposes without the burden of extensive data labeling and retraining.