FOLLOWIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

Abstract

Modern Language Models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. While Information Retrieval (IR) models use these LMs as the backbone of their architectures, virtually none of them allow users to provide detailed instructions alongside queries, thus limiting their ability to satisfy complex information needs. In this work, we study the use of instructions in IR systems. We build FOLLOWIR, a rigorous instruction evaluation benchmark for following real-world instructions in IR. FOLLOWIR repurposes detailed instructions-also known as narratives-developed for professional assessors to evaluate retrieval systems. In particular, we build our benchmark from three collections curated for shared tasks at the Text REtrieval Conference (TREC). Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework. Our results indicate that existing retrieval models fail to correctly use instructions, using them for basic keywords and struggling to understand long-form information. However, we show that it is possible for IR models to learn to follow complex instructions: our new FOLLOWIR-7B model has significant improvements after fine-tuning on our training set.