MT R: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning

Abstract

A long-standing difficulty in AI is the introduction of human-like reasoning in machine reading comprehension. Since algorithmic models can already perform as well as humans on simple quality assurance tasks thanks to the development of deep learning techniques, more difficult reasoning datasets have been presented. However, these datasets mainly focus on a single type of reasoning. There are still significant gaps in the studies when compared to the complex reasoning used in daily life because we can mix and match different types of reasoning unconsciously. In this work, we introduce a brand-new dataset, named MT R. There are two subsets of it: (1)the first is mainly used to explore mixed reasoning abilities and combines deductive and inductive reasoning; (2)the second integrates inductive and defeasible reasoning for detecting non-monotonic reasoning ability. It consists of more than 30k instances, requiring models to infer relations between characters in short stories. Compared with the corresponding single reasoning datasets, MT R serves as a more challenging one, highlighting the gap in language models' ability to handle sophisticated inference.