Distant Supervision in BERT-based Adhoc Document Retrieval

Abstract

Recently introduced pre-trained contextualized autoregressive models like BERT have shown improvements in document retrieval tasks. One of the major limitations of the current approaches can be attributed to the manner they deal with variable-size document lengths using a fixed input BERT model. Common approaches either truncate or split longer documents into small sentences/passages and subsequently label them - using the original document label or from another externally trained model. The other problem is the scarcity of labelled query-document pairs that directly hampers the performance of modern data hungry neural models. This process gets even more complicated with the partially labelled large dataset of queries derived from query logs (TREC-DL). In this paper, we handle both the issues simultaneously and introduce passage level weak supervision in contrast to standard document level supervision. We conduct a preliminary study on the document to passage label transfer and influence of unlabelled documents on the performance of adhoc document retrieval. We observe that direct transfer of relevance labels from documents to passages introduces label noise that strongly affects retrieval effectiveness. We propose a weak-supervision based transfer passage labelling scheme that helps in performance improvement and gathering relevant passages from unlabelled documents.