Fair Server Selection in Edge Computing With Q-Value-Normalized Action-Suppressed Quadruple Q-Learning

Abstract

Edge computing is a promising paradigm that brings servers closer to users, leading to lower latencies and enabling latency-sensitive applications such as cloud gaming, virtual/augmented reality, telepresence, and telecollaboration. Due to the high number of possible edge servers and incoming user requests, the optimum choice of user-server matching has become a difficult challenge, especially in the 5G era where the network can offer very low latencies. In this article, we introduce the problem of fair server selection as not only complying with an application's latency threshold but also reducing the variance of the latency among users in the same session. Due to the dynamic and rapidly evolving nature of such an environment and the capacity limitation of the servers, we propose as solution a reinforcement learning (RL) method in the form of a quadruple Q-Learning model with action suppression, Q-value normalization, and a reward function that minimizes the variance of the latency. Our evaluations in the context of a cloud gaming application show that, compared to an existing methods, our proposed method not only better meets the application's latency threshold but is also more fair with a reduction of up to 35% in the standard deviation of the latencies experienced by users. Impact Statement-Technologically, the work impacts cloud gaming providers, although providers of any multiuser real-time collaborative application can also benefit. This work enables these providers to optimize their offerings with a higher quality of user experience than what is possible today: better latency compliance and lower latency variation among users in the same online session. Economically, this work can contribute to the market expansion of such services because it will provide users with lower latencies and a fairer opportunity to participate, which is synonymous with more customers because failure to meet the latency threshold of games is one of the main reasons for subscriber turnover. The technical contributions are twofold: we propose an RL approach for cloud/edge server selection considering the variance of latency, and not just latency as in existing methods, for fairer user experience. We also introduce action suppression, Quadruple Q-Learning (QQL), and Q-value normalization in RL.