Low complexity neural network equalizer for nonlinearity mitigation in digital subcarrier multiplexing systems

Abstract

This article conducts a comparative study of the complexity reduction of neural network (NN) models for nonlinearity compensation used in digital subcarrier multiplexing (DSCM)-based optical communication systems. We employ the NN model based on bi-directional long short-term memory (biLSTM) and 1D-convolutional NN (1D-CNN) layers. To reduce the computation complexity of the proposed solution, weight clustering is applied to the NN. We specifically compare the performance of our proposed NN-based equalizer with traditional methods such as chromatic dispersion compensation (CDC), digital back-propagation (DBP) and alternative NN, based on triplet coefficients in perturbation analysis, proposed in previous literature, evaluating both Q-factor performance and computational complexity. Our findings show that the NN-based equalizer offers competitive Q-factor improvements while significantly reducing computational complexity, particularly when weight clustering is used. We show that the complexity of the NN can be reduced by up to 91.1% compared to the NN based on the perturbation analysis proposed in previous literature and by 31.5% compared to the DBP with 1 step per span. The results underscore the potential of NN-based approaches to deliver high-performance nonlinear compensation with lower computational demands, positioning them as a promising solution for future optical nonlinear equalizers.