Review on Modulation Formats and Channel Coding in Free Space Optical Communication: Selection Criteria, Trade-Offs, and Emerging Trends

Abstract

Free Space Optical (FSO) communication systems provide extremely large modulation bandwidths in the optical spectrum (hundreds of THz), enabling potential data rates from hundreds of Mbps to multiple Tbps, far exceeding congested radio frequency (RF) bands. In practical deployments, however, achievable throughput is limited by atmospheric impairments, including turbulence-induced fading, weather-dependent attenuation, beam divergence, and precise pointing requirements. Motivating the need for adaptive optics, hybrid FSO/RF architectures, advanced modulation techniques, and forward error correction coding to maintain reliable high-throughput communication. Nevertheless, FSO links can still provide substantially higher capacity than RF links, particularly in space, due to their much higher carrier frequencies and wider available spectrum. FSO systems span terrestrial, aerial, and space applications, with mitigation strategies tailored to factors such as system complexity, latency, supported applications, power constraints, and size/weight limitations. This survey presents a comprehensive analysis of modulation and channel coding schemes across these scenarios, providing quantitative comparisons previously unavailable. We review key channel impairments, system limitations from experimental studies, and evaluate trade-offs for modulation formats and coding schemes using metrics including bit error rate, outage probability, power and spectral efficiency, coding gain, and implementation complexity. Moreover, we discuss emerging trends and future directions, including AI-driven adaptive modulation and coding, orbital angular momentum multiplexing, and intelligent distributed multi-node systems. This survey serves as both a comprehensive reference and a practical design guide, offering actionable recommendations that map modulation and coding strategies to specific platforms, from terrestrial and uncrewed aerial vehicle (UAV) networks to high-altitude platforms, satellite, and deep-space systems.