A Multimodal Transformer-Based Fusion Model for Enhanced Relations Extraction From Texts

Abstract

Transformer models are significantly advancing the field of natural language processing (NLP), particularly in tasks such as the extraction of concepts and relationships from texts. However, while various studies employ single transformer models for one or both tasks and achieve performance improvements, there remain some limitations. A single model may struggle to optimize for both tasks simultaneously, resulting in poor performance in one or both areas. Furthermore, a single model may experience increased computational and memory demands when used for both tasks, as it must process and retain both entity-level and relation-level features. This can lead to inefficiencies and poor performance, especially on large or complex datasets. Moreover, when a single transformer model handles both tasks, errors in concept extraction can directly affect relation extraction, as the latter relies on accurate identification of entities. To address these limitations, this study introduces a novel Multimodal Transformer-based Fusion (MTF) model that integrates token-level representations and entity representations from different transformer models for enhanced relation classification. The proposed MTF model employs a fused cross-attention mechanism to effectively integrate these heterogeneous representations within the transformer encoder, leveraging complementary linguistic and entity-specific information. Through a structured tokenization process, we generate unified token and entity embeddings, enabling the model to learn a more distinctive representation space for improved relational understanding. The proposed MTF model is evaluated on five transformer-based architectures using the Conll04 and NYT benchmark datasets. The transformer-based architectures include three models for concept extration, namely, BERT, DistilBERT and RoBERTa, and two models for relation extraction, namely, LUKE and mLUKE. The comparison of individual model results reveal RoBERTa and mLuke as the best performing models for concept extraction and relation extraction, respectively. On both tasks, fusing the best two performing models results in the proposed MTF model outperforming the best individual models in terms of precision, recall, and F1 score. Furthermore, a comparative analysis of the results with related studies, shows the superior performance of the proposed MTF model with higher accuracy, precision, recall, and F1-score of 96.86%, 96.89%, 96.86%, 96.87%, respectively.