Actor-Critic Reinforcement Learning for Throughput-Optimized Power Allocation in Energy Harvesting NOMA Relay-Assisted Networks

Abstract

In fifth-generation (5G) and beyond (B5G) communication systems, the growing number of connected devices and the increased traffic on the network lead to substantial energy consumption, which requires energy-efficient and high-speed communication solutions. Integrating non-orthogonal multiple access (NOMA), energy harvesting (EH), and millimeter wave (mmWave) technologies has emerged as a powerful approach for achieving massive connectivity and energy-efficient communication paradigms. NOMA-based relay-assisted mmWave networks offer high directivity and enhanced data throughput. However, their design faces significant challenges, such as blockage, limited range, Line-of-Sight (LOS) constraints, and uncertainties in channel gain. Integrating EH and NOMA brings design constraints, namely the uncertainty and dynamic nature of EH sources, that complicate energy management and NOMA's power multiplexing challenges in optimizing power allocation. These factors require optimizing power and resources to ensure seamless connectivity and energy efficiency. Traditional optimization methods face challenges due to uncertainties in channel gains, EH, and blockages. Although reinforcement learning (RL) is typically used to manage uncertain environments, conventional RL algorithms cannot handle such environments with infinite state and action spaces. To address these challenges, this paper proposes a novel power-allocation framework that integrates an EH-capable source node, a relay, and multiple power-domain NOMA-based users. The proposed framework has two phases. During the first phase, the energy-harvesting source communicates with the relay to maximize the data rate while learning an optimal power allocation policy using an actor-critic approach. This method adapts to the uncertain EH process and varying channel conditions while addressing the limitations associated with infinite state and action spaces inherent in traditional RL for optimal power allocation. The second phase consists of a NOMA-based power allocation mechanism that assigns different powers to the users, such that the data received at the relay are transmitted to its designated users. As it turns out, this problem is non-convex. Hence, we use the sequential convex approximation method to solve this problem. Simulation results demonstrate that the proposed framework significantly outperforms traditional power allocation frameworks in data rate maximization and energy efficiency.