Lending an Ear: How LLMs Hear Your Banking Intentions

Abstract

Traditional language models in NLP require a considerable amount of labeled examples, which is not always available in data-limited domains like banking. Large Language Models (LLMs) are known to perform effectively with few-shot learning in various domains with just 1-5 examples per class. However, the use of open-source instruction-tuned LLMs, and how they compare to closed-source LLMs and modern few-shot learning approaches like contrastive learning in data-constrained use-cases has been under-explored. Additionally, the understanding of performance-cost trade-offs of these methods, as well as the consideration of infrastructure resource-limited settings through optimal usage of smaller versions of LLMs (7B-9B parameters), a critical concern for budget-limited organizations, has not been studied comprehensively. Our work addresses these gaps by studying the aforementioned approaches over the Banking77 financial intent detection dataset, including the evaluation and comparison of cutting-edge LLMs by Meta, Google, Mistral-AI, OpenAI, and Anthropic in a comprehensive set of few-shot scenarios which include examples selected by a human-expert, as well as with a cost-effective querying method based on retrieval-augmented generation (RAG). We observed that smaller open-source LLMs are able to out-perform larger closed-source ones with effective prompts and RAG. Moreover, they offer a significantly better performance-cost ratio than their larger closed-source counterparts. We also experiment with data-augmentation by using LLMs to generate artificial labeled examples, which is able to improve performance slightly in a data-scarce scenario. Finally, we explored benefits of fine-tuning using three parameter efficient methods and propose BAI-Fintent, an LLM based on fine-tuned Mistral-7B, that out-performs all other approaches at customer banking intent identification.