A Model of Unsupervised Formal Learning for Natural Language

Abstract

Formal language theory has shown that strong notions of learnability do not apply directly to classes of formal language that may include natural languages, regardless of the number of strings presented to the learner. However, research in statistical learning has shown that neural network language models can predict certain linguistic properties when trained on text alone. Unsupervised statistical models that leverage grammar formalisms have not yet achieved the same level of performance, despite advantages in interpretability. I introduce a variably supervised learning algorithms that can directly apply the rules of a number of different grammar formalisms to learn from natural language texts. This algorithm, the Missing Link algorithm, allows for the direct comparison of different grammar formalisms and learning environments. I compare results for Combinatory Categorial Grammars, Tree-Adjoining Grammars, and Relational Grammars with a number of different parameters. These results show that weakly equivalent grammar formalisms, such as CCGs and TAGs, perform differently in unsupervised learning and that directly encoding linguistic features into Relational Grammars can improve performance at specific linguistic tasks, at least in English. I also show that the same algorithm can be used to learn simple a semantics semantics using Abstract Meaning Representations. These results open new lines of research into the exploration of learning models for natural language.