Incorporating Label Co-Occurrence into Neural Network-Based Models for Multi-Label Text Classification

Abstract

Multi-label text classification (MLTC) addresses a fundamental problem in natural language processing, which assigns multiple relevant labels to each document. In recent years, Neural Network-based models (NN models) for MLTC have attracted much attention. In addition, NN models achieve favorable performances because they can exploit label correlations in the penultimate layer. To further capture and explore label correlations, we propose a novel initialization to incorporate label co-occurrence into NN models. First, we represent each class as a column vector of the weight matrix in the penultimate layer, which we name the class embedding matrix. Second, we deduce an equation for correlating the class embedding matrix with the label co-occurrence matrix, ensuring that relevant classes are denoted by vectors with large correlations. Finally, we provide a theoretical analysis of the equation, and propose an algorithm to calculate the initial values of the class embedding matrix from the label co-occurrence matrix. We evaluate our approach with various text extractors, such as Recurrent Neural Network (RNN), Convolutional Neural Network (CNN) and Transformer on four public datasets. The experimental results demonstrate that our approach markedly improves the performance of existing NN models.