McGill BabyLM Shared Task Submission: The Effects of Data Formatting and Structural Biases

Abstract

In this study, we describe our submission to the 2023 BabyLM shared-task’s strict-small track. Our findings demonstrate the feasibility of training high-performing models within the constraints of limited data, computational resources, and time. We provide evidence that the formatting of input can significantly impact downstream performance. Furthermore, the induction of structural biases into the models through the use of part-of-speech trees yields modest benefits. Our most successful model achieves 79% on the BLiMP evaluations and 72% on the SuperGLUE evaluations. All models trained during this study can be found at https://huggingface.co/mcgillbabylm.