Knowledge Base Index Compression via Dimensionality and Precision Reduction

2Citations
Citations of this article
32Readers
Mendeley users who have this article in their library.

Abstract

Recently neural network based approaches to knowledge-intensive NLP tasks, such as question answering, started to rely heavily on the combination of neural retrievers and readers. Retrieval is typically performed over a large textual knowledge base (KB) which requires significant memory and compute resources, especially when scaled up. On HotpotQA we systematically investigate reducing the size of the KB index by means of dimensionality (sparse random projections, PCA, autoencoders) and numerical precision reduction. Our results show that PCA is an easy solution that requires very little data and is only slightly worse than autoencoders, which are less stable. All methods are sensitive to pre- and post-processing and data should always be centered and normalized both before and after dimension reduction. Finally, we show that it is possible to combine PCA with using 1bit per dimension. Overall we achieve (1) 100× compression with 75%, and (2) 24× compression with 92% original retrieval performance.

Cite

CITATION STYLE

APA

Zouhar, V., Mosbach, M., Zhang, M., & Klakow, D. (2022). Knowledge Base Index Compression via Dimensionality and Precision Reduction. In Spa-NLP 2022 - 1st Workshop on Semiparametric Methods in NLP: Decoupling Logic from Knowledge, Proceedings of the Workshop (pp. 41–53). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.spanlp-1.5

Register to see more suggestions

Mendeley helps you to discover research relevant for your work.

Already have an account?

Save time finding and organizing research with Mendeley

Sign up for free