Multi-modal ancient scripts recognition via deep learning with data homogenization and augmentation

Abstract

Ancient scripts provide invaluable insights into ancient societies, and their effective recognition is crucial for cultural relic preservation, textual decipherment, and heritage. Current research primarily focuses on single mode ancient text data recognition such as processing rubbings or handwritten scripts independently, yet ancient scripts exhibit diverse forms across modalities. To address this, we propose a novel multi-modal recognition framework capable of processing hybrid inputs like rubbings of oracle bone inscriptions and handwritten scripts. Our method employs two additional modules, a cross-modal data homogenization module to unify heterogeneous data representations and a data augmentation module to enhance model robustness, then achieve the recognition with convolutional neural networks. Evaluated on oracle bone inscriptions and bronze inscriptions datasets, our approach outperforms baseline methods in recognition accuracy and generalization capability across modalities.