Lexical Diversity and Syntactic Complexity in AI-Translated Legislative Texts
DOI:
https://doi.org/10.17507/tpls.1509.08Keywords:
lexical diversity, syntactic complexity, AI, legislative textsAbstract
This study investigates the performance of artificial intelligence (AI) in the translation of legislative texts, focusing on the quality of translations produced by ChatGPT 4o and DeepL Pro. By using TAALED and NeoSCA, we evaluated and compared a number of indices in lexical diversity and syntactic complexity of AI-generated and human translations of twenty Chinese legislative texts. We used JASP to calculate Bayes Factor and then compare the translations of human and AI. Our findings indicate that while AI models demonstrate notable strengths in function words diversity and coordinate syntactic structures, they still lag behind human translators in overall lexical diversity and syntactic complexity. The study underscores the potential and limitations of AI in legal translation, highlighting the necessity for human-AI collaboration to achieve high-quality translations in this specialized field.
References
Bergmann, R., Ludbrook, J., & Spooren, W. P. (2000). Different outcomes of the Wilcoxon—Mann—Whitney test from different statistics packages. The American Statistician, 54(1), 72-77.
Choudhury, R., & McConnell, B. (2013). Translation technology landscape report. TAUS BV, DeRijp.
Cooper, T. C. (1976). Measuring written syntactic patterns of second language learners of German. The Journal of Educational Research, 69(5), 176-183.
Du, A. (2023). Exploring the value spaces of human translation in the ChatGPT era and the transitions needed for translation education. Foreign Languages and Cultures, 7(4), 90-103. Doi: 10.19967/j.cnki.flc.2023.04.009.
Herbold, S., Hautli-Janisz, A., Heuer, U., Kikteva, Z., & Trautsch, A. (2023). A large-scale comparison of human-written versus ChatGPT-generated essays. Scientific Reports, 13(1), 18617.
JASP Team. (2024). JASP (version 0.18.3) [Computer software]. https://jasp-stats.org/download/
Jiao, W., Wang, W., Huang, J. T., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv. org. https://arxiv.org/abs/2301.08745
Johnson, W. (1944). Studies in language behavior: A program of research. Psychological Monographs, 56(2), 1-15.
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795. https://doi.org/10.1080/01621459.1995.10476572
Kyle, K., Crossley, S. A., & Jarvis, S. (2021). Assessing the validity of lexical diversity using direct judgements. Language Assessment Quarterly, 18(2), 154-170. https://doi.org/10.1080/15434303.2020.1844205
Lin, X., M. Afzaal, & Aldayel, H. S. (2023). Syntactic complexity in legal translated texts and the use of plain English: a corpus-based study. Humanities and Social Sciences Communications, 10(1), 1-9.
Lu, X. (2010). “Automatic analysis of syntactic complexity in second language writing.” International Journal of Corpus Linguistics, 15(4), 474-496. https://doi.org/10.1075/ijcl.15.4.02lu
Masson, M. E. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing. Behavior Research Methods, 43, 679-690.
McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381-392.
Moneus, A. M., & Sahari, Y. (2024). Artificial intelligence and human translation: A contrastive study based on legal texts. Heliyon, 10(6), 1-14.
Nunes Vieira, L., & Alonso, E. (2020). Translating perceptions and managing expectations: an analysis of management and production perspectives on machine translation. Perspectives, 28(2), 163–184. https://doi.org/10.1080/0907676X.2019.1646776
Prieto Ramos, F. (2024). Revisiting translator competence in the age of artificial intelligence: the case of legal and institutional translation. The Interpreter and Translator Trainer, 18(2), 148-173.
Scott, J. R. (2019). Legal Translation Outsourced. London: Oxford Studies in Language and Law.
Schad, D. J., Betancourt, M., & Vasishth, S. (2021). Toward a principled Bayesian workflow in cognitive science. Psychological Methods, 26(1), 103-126.
Tan, L. (2024). NeoSCA (version 0.1.4) [Computer software]. GitHub. https://github.com/tanloong/neosca
Wang, H., & Ma, K. (2023). The application of artificial intelligence in biomedical text translation: A comparative study. Chinese Science & Technology Translators Journal, 36(3), 23-26. https://doi:10.16024/j.cnki.issn1002-0489.2023.03.001.
Wang, L., & Wang, X. (2023). The study on the training model of machine translation post-editing competence in the era of ChatGPT. Technology Enhanced Foreign Language Education, (4), 16-23+115.
Wang, Z., & Mao, C. (2023). Quality evaluation and improvement of ChatGPT translation——A case study on ceramics texts. Shandong Ceramics, 46(4), 20-27.
Wen, X., & Tian, Y. (2024). The effectiveness of ChatGPT in translating China-specific discourse text. Shanghai Journal of Translation, (2), 27-34+94-95.
Yu, L. (2024). Lexical diversity and syntactic complexity in ChatGPT translation. Foreign Language Teaching and Research, 56(2), 297-307+321. Doi: 10.19923/j.cnki.fltr.2024.02.005.
Yuan, Y. (2023). Beyond Chatbots and towards artificial general intelligence (AGI): the success of ChatGPT and its implications for linguistics. Contemporary Linguistics, 25(5), 633-652.
Zhang, F., & Qu, X. (2021). Ethical review of legal translation technology. Foreign Languages in China, 18(6), 17-22.
