LoRA-Based Fine-Tuning of Local LLMs for Hallucination Detection in Indonesian RAG Systems

Abstract

—Retrieval Augmented Generation (RAG) improves the factual grounding of Large Language Models (LLMs) by incorporating external knowledge. However, RAG systems may still generate hallucinated responses, and this issue remains underexplored in Indonesian language settings, particularly in settings where local deployment is preferred. This study proposes a hallucination detection approach for Indonesian RAG systems using Low Rank Adaptation (LoRA) fine-tuning. To support this objective, the study constructs a dataset in the Human-Computer Interaction domain consisting of 908 context, question, and answer pairs. The dataset is classified into four categories: FACT-H, FAITH-H, LOG-H, and FAITHFUL. Three local LLMs, namely, Gemma-7B-it, LlaMA-2-7B chat, and Phi-3-medium-4k-instruct, were evaluated using 5-fold cross-validation. The results show that Gemma-7B-it achieved the best performance in the four-class setting, with a Macro F1 score of 0.846. In the binary classification setting, Gemma achieved an accuracy of 98.1 per cent. Further analysis shows that Gemma was particularly effective in recognizing FAITHFUL, FAITH-H, and FACT-H, while LOG-H remained the most difficult class to distinguish consistently.