Page segmentation using convolutional neural network and graphical model

Abstract

Page segmentation of document images remains a challenge due to complex layout and heterogeneous image contents. Existing deep learning based methods usually follow the general semantic segmentation or object detection frameworks, without plentiful exploration of document image characteristics. In this paper, we propose an effective method for page segmentation using convolutional neural network (CNN) and graphical model, where the CNN is powerful for extracting visual features and the graphical model explores the relationship (spatial context) between visual primitives and regions. A page image is represented as a graph whose nodes represent the primitives and edges represent the relationships between neighboring primitives. We consider two types of graphical models: graph attention network (GAT) and conditional random field (CRF). Using a convolutional feature pyramid network (FPN) for feature extraction, its parameters can be estimated jointly with the GAT. The CRF can be used for joint prediction of primitive labels, and combined with the CNN and GAT. Experimental results on the PubLayNet dataset show that our method can extract various page regions with precise boundaries. The comparison of different configurations show that GAT improves the performance when using shallow backbone CNN, but the improvement with deep backbone CNN is not evident, while CRF is always effective to improve, even when combining on top of GAT.