Multisensory Integration in Lexical Processing: Predicting Word Recognition Through Cross-Modal Capability
DOI:
https://doi.org/10.17507/tpls.1411.22Keywords:
multisensory integration, cross-modal capability, lexical processing, word recognitionAbstract
The integration of multisensory in lexical processing has garnered significant interest in cognitive linguistics and psycholinguistics. This study investigates the predictive power of cross-modal capability (CMC) on word recognition, focusing on how sensory integration influences cognitive processing and lexical decision-making. By collecting five-sense ratings for synesthetic adjectives, we computed the cross-modal capability for each word. We then conducted regression analyses to examine the relationship between these indices and reaction time in lexical decision tasks. Our findings reveal that words with higher cross-modal capabilities, as measured by CMC, are recognized and processed more quickly. This superior predictive power of CMC underscores the importance of sensory associations in lexical processing and highlights the need for comprehensive indices that account for multiple sensory inputs. These results provide deeper insights into the cognitive mechanisms underlying word recognition and processing speed, suggesting that CMC is a better independent variable for partially predict lexical cognition and recognition. This research contributes to our understanding of multisensory integration in lexical processing and has implications for models of word recognition and cognitive linguistics.
References
Auvray, M., & Spence, C. (2008). The multisensory perception of flavor. Consciousness and Cognition, 17(3), 1016–1031.
Balota, D. A., Yap, M. J., & Hutchison, K. A. (2012). Megastudies: What do millions (or so) of trials tell us about lexical processing? In J. S. Adelman (Ed.), Visual word recognition: Models and methods, orthography and phonology (pp. 90–115). Psychology Press.
Barsalou, L. W. (2008). Grounded cognition. Annual Review of Psychology, 59, 617–645.
Bergen, B. K. (2012). Louder than words: The new science of how the mind makes meaning. New York: Basic Books.
Bernabeu, P. (2018). Dutch modality exclusivity norms for 336 properties and 411 concepts [Unpublished manuscript]. School of Humanities, Tilburg University, the Netherlands.
Bonin, P., Méot, A., Ferrand, L. et al. (2015). Sensory experience ratings (SERs) for 1,659 French words: Relationships with other psycholinguistic variables and visual word recognition. Behav Res, 47, 813–825.
Chedid, G., Brambati, S. M., Bedetti, C., Rey, A. E., Wilson, M. A., & Vallet, G. T. (2019). Visual and auditory perceptual strength norms for 3,596 French nouns and their relationship with other psycholinguistic variables. Behav. Res. Methods, 51, 2094–2105.
Chen, I. H., Zhao, Q., Long, Y., Lu, Q., & Huang, C. R. (2019). Mandarin Chinese modality exclusivity norms. PLoS ONE, 14(2), e0211336. https://doi. org/10.1371/journal.pone.0211336
Cytowic, R. E. (2002). Synesthesia: A union of the senses. Cambridge, MA: MIT press.
Cytowic, R. E., & Eagleman, D. (2009). Wednesday is indigo blue: Discovering the brain of synesthesia. Cambridge, MA: MIT Press.
Díez-Álamo, A.M., Díez, E., Wojcik, D.Z. et al. (2019). Sensory experience ratings for 5,500 Spanish words. Behav Res, 51, 1205–1215.
Juhasz, B.J. & Yap, M.J. (2013). Sensory experience ratings for over 5,000 mono- and disyllabic words. Behav Res, 45, 160–168.
Juhasz, B. J., Yap, M. J., Dicke, J., Taylor, S. C., & Gullick, M. M. (2011). Tangible Words are Recognized Faster: The Grounding of Meaning in Sensory and Perceptual Systems. Quarterly Journal of Experimental Psychology, 64(9), 1683–1691.
Lee, J. & Shin, J-A. (2023). The cross-linguistic comparison of perceptual strength norms for Korean, English and L2 English. Front. Psychol. 14:1188909. https://doi.org/10.3389/fpsyg.2023.1188909
Levinson, S. C., & Majid, A. (2014). Differential ineffability and the senses. Mind & Language, 29, 407–427.
Lynott, D., & Connell, L. (2009). Modality exclusivity norms for 423 object properties. Behavior Research Methods, 41, 558–564.
Lynott, D., & Connell, L. (2013). Modality exclusivity norms for 400 nouns: The relationship between perceptual experience and surface word form. Behavior Research Methods, 45, 516–526.
Lynott D, Connell L, Brysbaert M, Brand J, & Carney J. (2020). The Lancaster Sensorimotor Norms: multidimensional measures of perceptual and action strength for 40,000 English words. Behav Res Methods, 52(3),1271–1291.
Miklashevsky, A. (2018). Perceptual experience norms for 506 Russian nouns: modality rating, spatial localization, manipulability, imageability and other variables. Journal of Psycholinguist. Res., 47, 641–661.
Morucci, P., Bottini, R., & Crepaldi, D. (2019). Augmented modality exclusivity norms for concrete and abstract Italian property words. Journal of Cognition, 2(1), 42.
Pecher, D., Zeelenberg, R., & Barsalou, L. W. (2003). Verifying properties from different modalities for concepts produces switching costs. Psychological Science, 14, 119–124.
Pexman, P. M., Siakaluk, P. D., & Yap, M. J. (2013). Introduction to the research topic meaning in mind: semantic richness effects in language processing. Front. Hum. Neurosci., 7, 723.
Rolls, E. T. (2015). Emotion and decision making explained. Oxford University Press.
Shapiro, L. (2019). Embodied Cognition (2nd ed.). Routledge.
Speed, L. J., & Majid, A. (2017). Dutch modality exclusivity norms: simulating perceptual modality in space. Behav. Res. Methods, 49, 2204–2218.
Speed, L.J., & Brybaert, M. (2022). Dutch sensory modality norms. Behav Res, 54, 1306–1318.
Vukovic, N., Fardo, F., & Shtyrov, Y. (2019). When words burn – language processing differentially modulates pain perception in typical and chronic pain populations. Language and Cognition, 10(4), 626–640.
Wang, L.Y., Li, C.L., Han, Z.T., Wu, Q., Sun, L.W., Zhang, X., Go, Ritsu, Wu, J.L., & Yan, T.Y. (2022). Spatiotemporal and sensory modality attention processing with domain-specific representations in frontoparietal areas. Cerebral Cortex, 32(24), 5489–5502.
Winter, B. (2016). Taste and smell words form an affectively loaded and emotionally flexible part of the English lexicon. Language, Cognition and Neuroscience, 31(8), 975–988.
Winter, B. (2019). Sensory Linguistics: Language, perception and metaphor. John Benjamins.
Wu, Chenggang & Mu, Xin. (2023). Sensory experience ratings (SERs) for 1,130 Chinese words: relationships with other semantic and lexical psycholinguistic variables. Linguistics Vanguard, 9(1), 151–159.
Yap, M. J., Pexman, P. M., Wellsby, M., Hargreaves, I. S., & Huff, M. J. (2012). An Abundance of Riches: Cross-Task Comparisons of Semantic Richness Effects in Visual Word Recognition. Frontiers in Human Neuroscience, 6, https://doi.org/10.3389/fnhum.2012.00072
Zhao, Q.Q., Xiong, J.J., & Huang, C.R. (2019). Tonggan, Yinyu yu Renzhi: Tonggan Xianxiang zai Hanyu zhong de Xitongxing Biaoxian yu Yuyanxue Jiazhi [Linguistic synaesthesia, metaphor, and cognition: The systematicity and significance of linguistic synaesthesia in Chinese]. Zhong guo yu wen, 2, 240-253.
