Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

3Citations
Citations of this article
8Readers
Mendeley users who have this article in their library.

Abstract

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset. We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that "optimal"subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.

References Powered by Scopus

A sequential algorithm for training text classifiers

1916Citations
N/AReaders
Get full text

Billion-Scale Similarity Search with GPUs

1735Citations
N/AReaders
Get full text

Active learning with statistical models

1431Citations
N/AReaders
Get full text

Cited by Powered by Scopus

Scaling Laws For Dense Retrieval

2Citations
N/AReaders
Get full text

Resisting the Lure of the Skyline: Grounding Practices in Active Learning for Morphological Inflection

1Citations
N/AReaders
Get full text

Simple Domain Adaptation for Sparse Retrievers

0Citations
N/AReaders
Get full text

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Cite

CITATION STYLE

APA

Althammer, S., Zuccon, G., Hofstätter, S., Verberne, S., & Hanbury, A. (2023). Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection. In SIGIR-AP 2023 - Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (pp. 139–149). Association for Computing Machinery, Inc. https://doi.org/10.1145/3624918.3625333

Readers over time

‘23‘2402468

Readers' Seniority

Tooltip

PhD / Post grad / Masters / Doc 4

100%

Readers' Discipline

Tooltip

Computer Science 4

80%

Medicine and Dentistry 1

20%

Save time finding and organizing research with Mendeley

Sign up for free
0