Enforcing Prediction Consistency Across Orthogonal Planes Significantly Improves Segmentation of FIB-SEM Image Volumes by 2D Neural Networks.

Abstract

Volume electron microscopy (vEM) is transforming cell biology by generating high-resolution 3D reconstructions of large biological samples. However, segmentation of specific features such as mitochondria from highly heterogenous image volumes remains a bottleneck-even powerful deep learning (DL) approaches reveal various limitations and artifacts [1]. Within vEM technologies, focused ion beam scanning electron microscopy (FIB-SEM) can yield isotropic-voxel data where information in orthogonal planes (xy, xz, yz) is essentially interchangeable; here, we exploit this to develop a two-step DL algorithm for segmentation. First, we train a DL model to segment specific features in 2D image slices. Crucially, to incorporate 3D context into the predicted segmentation of a target volume, we run inference over xy, xz and yz planes, and average the results at each voxel, a procedure we call "ortho-plane inference". In the second step, we use the target volume and predicted segmentation to train a new 2D model in a weakly supervised setting with "bootstrapping". Bootstrapping enforces prediction consistency between adjacent voxels of the same object regardless of viewing orientation. This two-step algorithm results in a 23% increase in Intersection-over-Union (IoU) over the best case scenarios for ortho-plane inference without bootstrapping and a 35% IoU increase over "2D stack" inference (Fig 2). Sampling 3D volumes while staying in a 2D regime makes this approach nimble and thus well suited to vEM researchers with limited image and compute resources. In DL, 2D models are the most memory and data efficient option for 3D image segmentation. A volume containing 100 cubic voxels that would be a single example for a 3D model generates 300 examples for a 2D model after slicing along the principal axes. This 2D model would also have roughly 3x fewer parameters and could readily be initialized with weights pretrained on ImageNet. To address the chief disadvantage of working in 2D, i.e. the loss of valuable 3D context, we incorporate 3D information through ortho-plane inference. This inference strategy results in improved performance but is diminished by two key weaknesses in the model: a susceptibility to small changes in object appearance between adjacent image slices and to larger changes between orthogonal slices. Common examples of these errors are the "stacked pancake" artifact, familiar to researchers in the vEM field, and "cross-hatching" patterns, shown in Figure 2. In this advance, we train a second dataset-specific neural network to learn the noise patterns associated with these errors and eliminate them.