Probing Relational Knowledge in Language Models via Word Analogies

Abstract

Understanding relational knowledge plays an integral part in natural language understanding. When it comes to pre-trained language models (PLMs), prior work has been focusing on probing relational knowledge by filling the blanks in pre-defined prompts such as "The capital of France is -". However, these probes may be affected by the co-occurrence of target relation words and entities (e.g. "capital", "France" and "Paris") in the pre-training corpus. In this work, we extend these probing methodologies leveraging analogical proportions as a proxy to probe relational knowledge in transformer-based PLMs without directly presenting the desired relation. In particular, we analysed the ability of PLMs to understand (1) the directionality of a given relation (e.g. Paris-France is not the same as France-Paris); (2) the ability to distinguish types on a given relation (both France and Japan are countries); and (3) the relation itself (Paris is the capital of France, but not Rome). Our results show how PLMs are extremely accurate at (1) and (2), but have room for improvement for (3). To better understand the reasons behind this behaviour and the types of mistake made by PLMs, we provide an extended quantitative analysis.