Delving Into the Influence of Visual Input on the Load-Related Silent Pauses During SI: What About Gestures?
DOI:
https://doi.org/10.17507/tpls.1406.16Keywords:
visual input, semantically related gestures, silent pauses, SI, cognitionAbstract
This research delves into the realm of simultaneous interpreting (SI) with a focus on the Portuguese-Chinese language pair, examining the interplay between visual inputs and cognitive load. This study posits that visual cues such as hand gestures may influence the cognitive load during SI, a topic that remains controversial in interpreting studies. To address this, we conducted an empirical study involving 18 trainee interpreters divided into two groups: a control group receiving only audio input and an experimental group with additional video input. Utilizing ELAN 6.3 software, we analyzed silent pauses exceeding 300ms to gauge the cognitive load. The research focused on how audio and video inputs impact these silent pauses, with a special emphasis on segments accompanied by semantically related hand gestures. The results revealed that the average duration of silent pauses was marginally shorter for interpreters with video input, although the differences between the two groups were not statistically significant. Intriguingly, for both groups, the duration of pauses significantly increased during segments with semantically related gestures, underscoring the inherent high cognitive demand of these segments, irrespective of visual input. A notable discovery was the marked increase in fluency for participants with visual access when interpreting segments accompanied by gestures, which suggests that semantically related gestures provide cognitive benefits. Overall, this study contributes to the ongoing discourse on the role of visual inputs in SI, highlighting the potential of gesture input to alleviate cognitive load and improve interpreter performance.
References
Anderson, L. (1979). Simultaneous interpretation: Contextual and translation aspects. https://doi.org/https://doi.org/10.1075/btl.3.11
Anderson, L. (1994). Simultaneous interpretation: Contextual and translation aspects. InS. Lambert & B. Moser-Mercer (Eds.), Bridging the gap: Empirical research in simultaneous interpretation. https://doi.org/https://doi.org/10.1075/btl.3.11
Baddeley, A. (2000). The episodic buffer: a new component of working memory? Trends in cognitive sciences, 4(11), 417-423. https://doi.org/https://doi.org/10.1016/S1364-6613(00)01538-2
Balzani, M. (1990). le contact visual en interpretation simultanée. Paper presented at the Aspects of Applied and Experimental Research on Conference Interpretation.
Baxter, R. N. (2016). Exploring the possible effects of visual presentations on synchronicity and lag in simultaneous interpreting. Sendebar, 27, 9-23. Retrieved June 30, 2023, from https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwje-qS7loyEAxVqLEQIHfhpD8YQFnoECA4QAQ&url=https%3A%2F%2Frevistaseug.ugr.es%2Findex.php%2Fsendebar%2Farticle%2Fdownload%2F3834%2F5052%2F11845&usg=AOvVaw0hu8OPaxB5gXWfGH8AP-YJ&opi=89978449
Bonnici, H. M., Richter, F. R., Yazar, Y., & Simons, J. S. (2016). Multimodal feature integration in the angular gyrus during episodic and semantic retrieval. Journal of Neuroscience, 36(20), 5462-5471. https://doi.org/10.1523/JNEUROSCI.4310-15.2016
Gieshoff, A. C. (2017). Audiovisual speech decreases the number of cognate translations in simultaneous interpreting. Empirical modelling of translation and interpreting, 7, 313. https://doi.org/10.5281/zenodo.1090978
Gieshoff, A. C. (2018). The impact of audio-visual speech on work-load in simultaneous interpreting. (Doctor Dissertation). Johannes Gutenberg-Universität Mainz, Mainz. JGU-Publikationen database.
Gieshoff, A. C. (2021a). Does it help to see the speaker’s lip movements? An investigation of cognitive load and mental effort in simultaneous interpreting. Translation, Cognition & Behavior. https://doi.org/10.1075/tcb.00049.gie
Gieshoff, A. C. (2021b). The impact of visible lip movements on silent pauses in simultaneous interpreting. Interpreting. https://doi.org/https://doi.org/10.1075/intp.00061.gie
Goldman-Eisler, F. (1958). The predictability of words in context and the length of pauses in speech. Language and Speech, 1(3), 226-231. https://doi.org/https://doi.org/10.1177/002383095800100308
Grömping, U. (2009). Variable importance assessment in regression: linear regression versus random forest. The American Statistician, 63(4), 308-319. https://doi.org/https://doi.org/10.1198/tast.2009.08199
Grosjean, F., & Deschamps, A. (1972). Analyse des variables temporelles du français spontané. Phonetica, 26(3), 129-156. https://doi.org/https://doi.org/10.1159/000259407
Han, L., Lu, J., Wen, Z. E., & Tian, Y. (2023). Momentary engagement in simultaneous versus consecutive interpreting: through the lens of translanguaging and CDST. https://doi.org/https://doi.org/10.3389/fpsyg.2023.1180379
Hieke, A. E., Kowal, S., & O'Connell, D. C. (1983). The trouble with" articulatory" pauses. Language and Speech, 26(3), 203-214. https://doi.org/https://doi.org/10.1177/002383098302600302
Jiang, X. L., & Jiang, Y. (2020). Effect of dependency distance of source text on disfluencies in interpreting. Lingua, 243, 18. https://doi.org/https://doi.org/10.1016/j.lingua.2020.102873
Kahng, J. (2014). Exploring utterance and cognitive fluency of L1 and L2 English speakers: Temporal measures and stimulated recall. Language Learning, 64(4), 809-854. https://doi.org/10.1111/lang.12084
Koelewijn, T., Bronkhorst, A., & Theeuwes, J. (2010). Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta psychologica, 134(3), 372-384. https://doi.org/10.1016/j.actpsy.2010.03.010
Lin, Y., Lv, Q., & Liang, J. (2018). Predicting fluency with language proficiency, working memory, and directionality in simultaneous interpreting. Frontiers in psychology, 9(1), 1543. https://doi.org/10.1016/j.plrev.2018.06
McKnight, P. E., & Najab, J. (2010). Mann‐Whitney U Test. The Corsini encyclopedia of psychology, 1-1. https://doi.org/ https://doi.org/10.1002/9780470479216.corpsy0524
Mead, P. (2000). Control of pauses by trainee interpreters in their A and B languages. The interpreters’ newsletter, 10(200), 89-102. Retrieved June 30, 2023, from https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=1a2485651848b1baeaed9cf12044976064d3a2a3
Mikkelson, H., & Jourdenais, R. (2015). The Routledge handbook of interpreting (H. Mikkelson & R. Jourdenais Eds.). London: Routledge New York.
Miller, J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14(2), 247-279. https://doi.org/10.1016/0010-0285(82)90010-x
Molholm, S., Sehatpour, P., Mehta, A. D., Shpaner, M., Gomez-Ramirez, M., Ortigue, S., . . . Foxe, J. J. (2006). Audio-visual multisensory integration in superior parietal lobule revealed by human intracranial recordings. Journal of neurophysiology, 96(2), 721-729. https://doi.org/10.1152/jn.00285.2006
Moser-Mercer, B. (2005). Remote interpreting: The crucial role of presence. Bulletin vals-asla, 81, 73-97. https://doi.org/https://benjamins.com/online/hts/articles/rem1/1000?filter-language=uk
Plevoets, K., & Defrancq, B. (2016). The effect of informational load on disfluencies in interpreting: A corpus-based regression analysis. Translation and Interpreting Studies. The Journal of the American Translation and Interpreting Studies Association, 11(2), 202-224. https://doi.org/10.1075/tis.11.2.04ple
Prandi, B. (2023). Computer-assisted simultaneous interpreting: A cognitive-experimental study on terminology (Vol. 22): Language Science Press.
Raupach, M. (2011). Temporal variables in first and second language speech production. In Temporal variables in speech (pp. 263-270): De Gruyter Mouton.
Rennert, S. (2008). Visual input in simultaneous interpreting. Meta: journal des traducteurs/Meta: Translators' Journal, 53(1), 204-217. https://doi.org/https://doi.org/10.7202/017983ar
Riggenbach, H. (1991). Toward an understanding of fluency: A microanalysis of nonnative speaker conversations. Discourse processes, 14(4), 423-441. https://doi.org/https://doi.org/10.1080/01638539109544795
Seeber, K. G. (2007). Thinking outside the cube: Modeling language processing tasks in a multiple resource paradigm. Paper presented at the Eighth Annual Conference of the International Speech Communication Association.
Seeber, K. G. (2011). Cognitive load in simultaneous interpreting: Existing theories—new models. Interpreting, 13(2), 176-204. https://doi.org/https://doi.org/10.1075/intp.13.2.02see
Seeber, K. G. (2012). Multimodal input in Simultaneous Interpreting: An eye-tracking experiment. Paper presented at the Proceedings of the 1st International Conference TRANSLATA, Translation & Interpreting Research: yesterday-today-tomorrow.
Seeber, K. G. (2017). Multimodal processing in simultaneous interpreting. The handbook of translation and cognition, 461-475. https://doi.org/https://doi.org/10.1002/9781119241485.ch25
Shreve, G. M., Lacruz, I., & Angelone, E. (2011). Sight translation and speech disfluency Performance analysis as a window to cognitive translation processes. In C. Alvstad, A. Hild, & E. Tiselius (Eds.), Methods and Strategies of Process Research: Integrative Approaches in Translation Studies (Vol. 94, pp. 93-120).
Skehan, P. (2003). Task-based instruction. Language teaching, 36(1), 1-14. https://doi.org/10.5296/ijl.v4i3.2203
Song, S. (2020). Fluency in simultaneous interpreting of trainee interpreters: the perspectives of cognitive, utterance and perceived fluency. (Ph.D.). Hong Kong Polytechnic University, Retrieved June 30, 2023, from https://theses.lib.polyu.edu.hk/bitstream/200/10418/1/991022378657903411.pdf
Song, S., & Li, D. (2020). The Predicting Power of Cognitive Fluency for the Development of Utterance Fluency in Simultaneous Interpreting. Frontiers in psychology, 11, 1864. https://doi.org/https://doi.org/10.3389/fpsyg.2020.01864
Sumby, W. H., & Pollack, I. (1954). Visual contribution to speech intelligibility in noise. The journal of the acoustical society of America, 26(2), 212-215. https://doi.org/https://doi.org/10.1121/1.1907309
Teder-Sälejärvi, W., McDonald, J., Di Russo, F., & Hillyard, S. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive Brain Research, 14(1), 106-114. https://doi.org/ 10.1016/s0926-6410(02)00065-4
Towell, R., Hawkins, R., & Bazergui, N. (1996). The development of fluency in advanced learners of French. Applied linguistics, 17(1), 84-119. https://doi.org/https://doi.org/10.1093/applin/17.1.84
Wang, B., & Li, T. (2015). An empirical study of pauses in Chinese-English simultaneous interpreting. Perspectives, 23(1), 124-142. Retrieved June 30, 2023, from https://www.polyu.edu.hk/edc/tdg/userfiles/file/42A5_paper2.pdf
Wang, X., & Wang, B. (2022). Identifying fluency parameters for a machine-learning-based automated interpreting assessment system. Perspectives, 1-17. Retrieved June 30, 2023, from https://doi.org/10.1080/0907676X.2022.2133618
Wickens, C. D. (2002). Multiple resources and performance prediction. Theoretical issues in ergonomics science, 3(2), 159-177. https://doi.org/https://doi.org/10.1080/14639220210123806
Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. (2006). ELAN: A professional framework for multimodality research. Paper presented at the 5th international conference on language resources and evaluation (LREC 2006).
Xin, S. (2020). ying han jiao ti chuan yi zhong bu dang ting dun de chu xian yuan yin he ying dui ce lue [Causes and coping strategies for the emergence of inappropriate pauses in English-Chinese consecutive interpreting]. (Unpublished master thesis). Shanghai International Studies University, Retrieved June 30, 2023, from 10.27316/d.cnki.gswyu.2020.000139
Yang, S. (2019). Investigating the Effect of Speech Rate on the Cognitive Load in Simultaneous Interpreting with Text. University of Macau, Retrieved June 30, 2023, from https://www.proquest.com/openview/150fd38d1a83c44ecd1e6f43671e2221/1.pdf?pq-origsite=gscholar&cbl=18750&diss=y
Yang, S., Li, D., & Lei, V. L. C. (2020). The impact of source text presence on simultaneous interpreting performance in fast speeches: Will it help trainees or not? Babel, 66(4-5), 588-603. https://doi.org/10.1075/babel.00189.yan
Zhao, N. (2022). Speech Disfluencies in Consecutive Interpreting by Student Interpreters: The Role of Language Proficiency, Working Memory, and Anxiety. Frontiers in psychology, 13. https://doi.org/10.3389/fpsyg.2022.881778
