Deep reinforcement learning paradigm for dense wireless networks in smart cities

Abstract

Wireless local area networks (WLANs) are widely deployed for Internet-centric data applications. Due to their extensive norm in our day-to-day wireless-enabled life, WLANs are expected to play a vital role for Internet of Things (IoT). It is predicted that by 2020, about 50 billion devices (things) will be connected via IoT. Consequently, WLANs need major improvements in both throughput and efficiency for such a massive device deployment in applications like smart offices, smart train stations, and smart stadiums for smart cities in IoT. New technologies continue to be introduced for WLAN applications for this purpose. The IEEE 802.11ac standard is the currently implemented amendment by the IEEE 802.11 standard working group that promises data rates at gigabits per second. The main features of the IEEE 802.11ac standard are adopting increased bandwidth and higher order modulation than the previous standards, and multiple-input, multiple-output (MIMO) and multiuser MIMO transmission modes. These features are designed to improve the user experience. In addition to technologies that enhance the efficiency of the WLAN, the IEEE 802.11ax High Efficiency WLAN (HEW) standard is also investigating and evaluating advanced wireless technologies to utilize the existing spectrum more efficiently. The next-generation dense WLAN, HEW is expected to confront ultradense user environments and radically new applications for smart cities. HEW is likely to provide four times higher network efficiency even in highly dense network deployments. However, the current WLAN itself faces huge challenge of efficient channel access due to its temporal-based MAC layer resource allocation (MAC-RA). WLAN uses a carrier sense multiple access with collision avoidance (CSMA/CA) procedure to access the channel resources, which is based on a binary exponential backoff (BEB) mechanism. In BEB, a random backoff value is generated from a contention window (CW) to obtain channel access. The CW size is doubled after every unsuccessful transmission and reset to its minimum value on successfully transmissions. However, this blindness when increasing and resetting the CW induces performance degradation. For a dense network, resetting the CW to its minimum size may result in more collisions and poor network performance. Likewise, for a small network environment, a blind increase in CW size may cause an unnecessarily long delay while accessing the channel. To satisfy the diverse requirements of dense WLANs, it is anticipated that prospective HEW will autonomously access the best channel resources with the assistance of sophisticated wireless channel condition inference in order to control channel collisions. Such intelligence is possible with the introduction of deep learning (DL) techniques in future WLANs. The potential applications of DL to the MAC layer of IEEE 802.11 standards have already been progressively acknowledged due to their novel features for future communications. Their new features challenge conventional communications theories with more sophisticated artificial intelligence-based theories. DL has been extensively proposed for the MAC layer of WLANs in various research areas, such as deployment of cognitive radio and communications networks. Deep reinforcement learning (DRL) is one of the DL techniques that are motivated by the behaviorist sensibility and control philosophy, where a learner can achieve an objective by interacting with the environment. In this chapter, a DRL-based intelligent paradigm is developed for channel access in dense WLANs in smart cities. One of the DRL models, Q-learning (QL), is used to propose an intelligent QL-based resource allocation (iQRA) mechanism for MAC-RA in dense wireless networks. iQRA exploits channel observation-based collision probability for network inference to dynamically and autonomously control the backoff parameters (such as backoff stages and CW values). The simulations performed in network simulator 3 (ns3) indicate that the proposed DRL-based iQRA paradigm learns diverse WLAN environments and optimizes its performance, compared to conventional non-intelligent MAC protocol, BEB. The performance of the proposed iQRA mechanism is evaluated in diverse WLAN network environments with throughput, channel access delay, and fairness as performance metrics.