Overview of the SISAP 2024 Indexing Challenge

Abstract

The SISAP 2024 Indexing Challenge invited replicable and competitive approximate similarity search solutions for datasets of up to 100 million real-valued vectors. Participants are evaluated on the search performance of their implementations under quality constraints. Using a subset of the deep features of a neural network model provided by the LAION-5B dataset, the challenge posed three tasks, each with its unique focus:Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries.Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1.Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. Task 1, Unrestricted indexing: Conduct a classical approximate nearest neighbors search, ensuring an average recall of at least 0.8 for 30-NN queries. Task 2, Memory-constrained indexing with reranking: Conduct nearest neighbors search in a low-memory setting where the dataset collection is only accessible on disk, ensuring the same quality as in Task 1. Task 3, Memory-constrained indexing without reranking: Conduct nearest neighbor search in a setting where the dataset cannot be accessed at search stage, ensuring an average recall of at least 0.4 for 30-NN queries. The present paper describes the details of the challenge, the evaluation system that was developed with it, and gives an overview of the submitted solutions.