Evaluating the Tradeoff Between Analogical Reasoning Ability and Efficiency in Large Language Models

Abstract

Recent advances in large language models (LLMs) have led to the general public's assumption of human-equivalent logic and cognition. However, the research community is inconclusive, especially concerning LLM's analogical reasoning abilities. Twenty-one proprietary and open-source LLMs were evaluated on two long-text/story analogy datasets. The LLMs produced mixed results on the four qualitative and seven quantitative metrics. LLMs performed well when tasked with determining the presence or absence of similar elements between stories based on the qualitative assessment of their outputs. However, despite this success, LLMs still struggled with the correct identification of the most analogous story to the base story. Further inspection indicates that the models struggled with recognizing high-order (similar to cause and effect) relationships associated with higher cognitive function(s). Regardless of the overall performance, there is a clear advantage that propriety has over open-source models concerning analogical reasoning. Last, this study suggests that LLM accuracy and their number of parameters explain over half of the variation in the energy consumed based on a statistically significant multivariate regression model. Future work may consider evaluating other types of reasoning and LLMs' learning abilities by providing 'correct' responses to guide future results.