Parameter-efficient fine-tuning approaches have recently garnered a lot of attention. Having considerably lower number of trainable weights, these methods can bring about scalability and computational effectiveness. In this paper, we look for optimal sub-networks and investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. Our empirical results suggest that every transformer module in BERT can act as a winning ticket: fine-tuning each specific module while keeping the rest of the network frozen can lead to comparable performance to the full fine-tuning. Among different modules, LayerNorms exhibit the best capacity for knowledge transfer with limited trainable weights, to the extent that, with only 0.003% of all parameters in the layer-wise analysis, they show acceptable performance on various target tasks. On the reasons behind their effectiveness, we argue that their notable performance could be attributed to their high-magnitude weights compared to that of the other modules in the pre-trained BERT. The code for this paper is freely available at https://github.com/m-tajari/transformer-transferability.
CITATION STYLE
AkbarTajari, M., Rajaee, S., & Pilehvar, M. T. (2022). An Empirical Study on the Transferability of Transformer Modules in Parameter-Efficient Fine-Tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 (pp. 10617–10625). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.emnlp-main.726
Mendeley helps you to discover research relevant for your work.