Consistency–accuracy correlation in hard-prompted LLMs for entity and relation extraction: empirical findings from plant-health data

Abstract

As large language models (LLMs) become increasingly popular for information extraction (IE), concerns persist regarding the stability and reliability of their outputs. While accuracy has traditionally been the main evaluation metric, consistency—defined as the stability of model outputs across repeated runs—has recently been proposed as a complementary signal of reliability. In this work, we examine the relationship between accuracy and consistency in hard-prompted generative LLMs applied to entity and relation extraction. We conduct a systematic evaluation using four LLMs (GPT, DeepSeek, Qwen, Kimi) on the EPOP corpus, a plant-health dataset with rich entity types, long-range relations, overlapping relations, and strong argument constraints. To refine the interpretation of consistency, we distinguish between recoverable output variations—those that preserve the meaning of the extracted information—and critical ones that result in semantic errors. Our results show that while some positive correlation between accuracy and consistency exists, it is model-dependent and varies with task complexity. In structured prediction tasks, we show that consistency should be measured at the semantic level, ignoring superficial variations in format or wording. These insights have important implications for using self-consistency as a confidence filter and for designing reliable generative IE pipelines in specialized domains.