Manuscripts Image Retrieval Using Deep Learning Incorporating a Variety of Fusion Levels

Abstract

The instantaneous search and retrieval of the most relevant images to a specific query image is a desirable application for all digital libraries. The automatic extraction and classification according to the most distinguishable features, is a crucial step to detect the similarities among images successfully. This study introduces a novel approach that utilizes a fusion model for classifying and retrieving historical Arabic manuscripts' images. To accomplish our goal, the images are first classified according to their extracted deep learning visual features utilizing a pre-trained convolutional neural network. Then, the texts written in the manuscripts' images are extracted and pre-processed to classify the images according to their textual features using an optimized bidirectional LSTM deep learning model with attention and batch normalization layers. Finally, both the visual and textual deep learning models are fused at three different fusion-levels named: decision-level, features-level, and score-level. The score-level fusion model resulted in a considerable improvement of each model used individually. Extensive experimentation and evaluation of the proposed fusion method on the collected ancient Arabic manuscripts dataset proved its robustness against other state-of-the-art methods recording 99% classification accuracy and 98% mean accuracy on the top-10 image retrieval.