Cognitive Plausibility in NLP

Summary

Language is a powerful tool of human communication that provides elegant mechanisms for expressing highly complex phenomena. We use language every day and in all aspects of our lives. The versatility and variability of language make it a difficult subject for computational modeling, in contrast to more systematic sensor signals. Language follows underlying rules only to surprise us with exceptions and ambiguities on all linguistic levels and understanding its subtleties requires even more culture-specific knowledge than interpreting images.

In spite of these complexities, humans usually process language effortlessly. We are able to vary our language use to smoothly adapt to the target audience and dynamically integrate situational cues for seamless disambiguation. Natural language processing (NLP) research has already spent decades trying to understand how to computationally model language but complex reasoning tasks and creative constructions still lead to obvious failures of models. Nevertheless, the success of the field is undeniable. It attracts a continuously growing number of researchers and language processing models have become a key technology in our daily lives. These developments are strongly linked to the increasing availability of large amounts of training data and more efficient computing resources. Neural language models (LMs) are trained on terabytes of data and optimize millions of parameters to extract patterns from text.

While the general benefit of computational modeling has been clearly established in multiple disciplines, we observe substantial disagreement in evaluating which properties characterize a ``useful'' model. we propose integrating cognitive plausibility as an additional factor and agree with their call for more multi-dimensional research to capture interactions. We think that a multilingual perspective is required to learn more about cognitively plausible principles of language processing. And cognitively more plausible models can lead to computationally more efficient models. Cognitive plausibility itself is a multi-faceted concept that varies considerably across disciplines. Computer scientists focus on the quantitative performance of the model, which should not be distinguishable from humans on static benchmark datasets. Neuroscientists focus on the biological plausibility of the model and aim to develop models of synaptic plasticity, which are evaluated on toy datasets that are much smaller than the common evaluation datasets in natural language processing. Psychologists focus on the plausibility of the learning processes in language models. They question the size and quality of the input data, examine memory and attention constraints, and explore learning curves. They try to isolate experimental factors by working with carefully designed stimuli that are often not representative of realistic language use. We approach the concept of cognitive plausibility by diving into interpretability research and identifying the potential of its methods for cognitively inspired research questions. In addition, we explore methods that integrate psycholinguistic data and theories into computational models.

Cooperators

Nora Hollenstein, Lena Jäger, Iza Škrjanec, Miyu Oba

Back to Research Overview.