Retrieval-Augmented Generation for Maritime Accident Report Analysis: Evaluating Large Language Models on Performance and Cybersecurity

Abstract

When accidents occur, official investigations are carried out, and reports are generated, which are usually reviewed for safety improvements. The retrieval of information is typically performed manually, which can lead to biases, errors, and poor judgement. Moreover, manual reviews can be tedious and highly time-consuming tasks. For these reasons, the implementation of LLMs has been analysed in this context. However, previous studies have been limited, and no proper justification for the implemented LLMs has been provided. Consequently, this work proposes a comparative framework to assess LLM candidates across two main dimensions: cybersecurity and performance. Specifically, a total of 9 LLMs from different providers were analysed, and 18 prompt injection techniques were implemented across 7 categories based on OWASP LLM01:2025 and previous academic studies. Additionally, a RAG system based on these results is introduced to validate the potential of these models in supporting experts in the retrieval of information from maritime accident reports. For validation purposes, a case study on the Marine Accident Investigation Branch (MAIB) reports was conducted. Results show that a comparative framework is required, as model selection may vary depending on the task being performed, which is critical from both a performance and cybersecurity perspective.