Fractional Gradient Optimizers for PyTorch: Enhancing GAN and BERT

Abstract

Machine learning is a branch of artificial intelligence that dates back more than 50 years. It is currently experiencing a boom in research and technological development. With the rise of machine learning, the need to propose improved optimizers has become more acute, leading to the search for new gradient-based optimizers. In this paper, the ancient concept of fractional derivatives has been applied to some optimizers available in PyTorch. A comparative study is presented to show how the fractional versions of gradient optimizers could improve their performance on generative adversarial networks (GAN) and natural language applications with Bidirectional Encoder Representations from Transformers (BERT). The results are encouraging for both state-of-the art algorithms, GAN and BERT, and open up the possibility of exploring further applications of fractional calculus in machine learning.