Abstracts Language, Literature, and Linguistic

Modeling and decoding semantic representations in language and vision

by Jerry https://orcid.org/0009-0005-7872-3529 Tang

Humans experience the world through many sources of information, including language and vision. This information is encoded in the brain as semantic representations, which provide an internal model of the world. In language, semantic representations enable us to understand the meanings of words and phrases. In vision, semantic representations enable us to recognize objects and actions. Because semantic representations are essential building blocks of cognition, understanding the semantic system is an important goal for basic and translational research. To study semantic representations, brain responses can be recorded while participants process naturalistic stimuli. Then, machine learning techniques can be used to model the relationship between the brain responses and the stimuli that evoked them. Two powerful machine learning techniques are encoding models and decoding models. Encoding models, which predict brain responses from stimulus features, can precisely characterize where and how concepts are represented in the brain. Decoding models, which predict stimulus features from brain responses, can demonstrate that a given concept is represented in a given brain area. Moreover, decoding models can enable brain-computer interfaces for people who struggle to move or speak. This dissertation describes four studies that model and decode semantic representations in language and vision. In the first study, I modeled how semantic representations relate to linguistic and visual experience, and found that many brain regions represent concepts by combining linguistic and visual information. In the second study, I compared semantic representations evoked by naturalistic stories and movies, and found that semantic representations are highly aligned across language and vision. In the third study, I developed an approach for decoding continuous language from semantic representations, and showed that decoders trained on brain responses during speech perception could be applied to brain responses during other linguistic and non-linguistic tasks. In the fourth study, I developed an approach for transferring semantic decoders across participants, and showed that language can be decoded from a participant without requiring any linguistic training data from that participant. Together, these studies provide new insights into the organization of the semantic system, and demonstrate how semantic representations can be decoded for applications such as speech restoration.