Is conversational AI ready for engineering licensure? A ChatGPT performance benchmark and prompting strategy evaluation

Abstract

The integration of artificial intelligence (AI) into specialized fields like civil engineering presents transformative opportunities. This study provides a comprehensive quantitative benchmark of the ChatGPT-4 model’s performance on the Fundamentals of Engineering (FE) Civil Exam. A two-phase methodology was employed. First, the model’s baseline performance was evaluated on a dataset of 100 representative exam questions using zero-shot prompting. Second, a follow-up experiment investigated the impact of advanced prompting strategies, including one-shot, on the 51 incorrectly answered questions. The initial findings reveal a significant performance disparity: ChatGPT-4 achieved high accuracy in conceptual domains like “Ethics and Professional Practice” (100%) but struggled in calculation-intensive areas such as “Statics” (14%), with an overall accuracy of 49%. The subsequent prompt engineering experiment, however, demonstrated that providing a single example (one-shot prompting) was the most effective strategy, correctly answering 30 of the 51 previously failed questions. These combined results offer critical evidence that while current LLMs have inherent limitations in analytical reasoning, their effectiveness can be substantially enhanced through strategic user interaction. The study concludes that AI should be implemented as a powerful supplemental tool, with an educational focus on teaching students how to effectively guide these models to achieve desired outcomes.