A Comparative Study on the Quality of English-Chinese Translation of Legal Texts Between ChatGPT and Neural Machine Translation Systems
DOI:
https://doi.org/10.17507/tpls.1409.18Keywords:
ChatGPT, legal translation, Neural Machine TranslationAbstract
This study conducts a comparative analysis of the quality of English-to-Chinese (E-C) and Chinese-to-English (C-E) translation of legal texts between Chat Generative Pre-trained Transformer (ChatGPT) and four online Neural Machine Translation (NMT) systems. The analysis includes both quantitative and qualitative evaluations. The results suggest that both ChatGPT and the NMT systems achieve satisfactory performance in translating legal texts from Chinese to English. Although the quality of ChatGPT’s C-E legal translation is slightly lower than that of the NMT systems, the difference is not statistically significant. However, neither ChatGPT nor the NMT systems meet a passing standard for E-C translation of legal texts, with the NMT systems showing better overall performance. Overall, ChatGPT and the NMT systems perform better at translating legal texts from Chinese to English compared to E-C translation. For E-C legal translation, ChatGPT’s quality is lower compared to the NMT systems. While the types of errors are similar in both systems, ChatGPT tends to exhibit more errors, some of which are more severe. This study serves as a reference for those choosing translation tools for E-C and C-E legal texts.
References
Cady, L., Tsou, B. K., & Lee, J. S. (2023). Comparing Chinese-English MT Performance Involving ChatGPT and MT Providers and the Efficacy of AI mediated Post-Editing. In Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track (pp. 205-216).
Feng, Z., & Zhang, D. (2022). Machine Translation and Human Translation Boost Each Other. Journal of Foreign Languages, (06), 77-87.
Grimm, D. R., Lee, Y. J., Hu, K., Liu, L., Garcia, O., Balakrishnan, K., & Ayoub, N. F. (2024). The utility of ChatGPT as a generative medical translator. European Archives of Oto-Rhino-Laryngology, 1-5.
Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., ... & Awadalla, H. H. (2023). How good are GPT models at machine translation? A comprehensive evaluation. arXiv preprint arXiv:2302.09210.
Jiao, W., Wang, W., Huang, J. T., Wang, X., Shi, S., & Tu, Z. (2023). Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv preprint arXiv:2301.08745.
Khoshafah, F. (2023). ChatGPT for Arabic-English translation: Evaluating the accuracy.
Lee, T. K. (2023). Artificial intelligence and posthumanist translation: ChatGPT versus the translator. Applied Linguistics Review. https://doi.org/10.1515/applirev-2023-0122.
Li, F. (2021). A Comparative Study on the Quality of English-Chinese and Chinese-English Translations in Neural Network-Based Online Machine Translation Systems. Shanghai Journal of Translators, (04), 46-52.
Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311-318).
Retrieved October 2023, from https://flk.npc.gov.cn/detail2.html?ZmY4MDgxODE3OTZhNjM2YTAxNzk4MjJhMTk2NDBjOTI%3D
Retrieved October 2023, from https://www.elegislation.gov.hk/hk/cap212
Retrieved October 2023, from https://www.elegislation.gov.hk/hk/cap212!sc?INDEX_CS=N
Retrieved October 2023, from https://www.pkulaw.com/en_law/39c1b78830b970eabdfb.html
Sanz-Valdivieso, L., & López-Arroyo, B. (2023). Google Translate vs. ChatGPT: Can non-language professionals trust them for specialized translation? In International Conference Human-informed Translation and Interpreting Technology (HiT-IT 2023) (pp. 97-107).
Wang, J., & Wen, Q. (2010). A Review of Domestic and International Automated Machine Scoring Systems: Implications for Automated Translation Scoring Systems for Chinese Students. Foreign Language World, (1), 75-81+91.
Wang, Z., & Mao, C. (2023). Assessing and Improving the Quality of ChatGPT Translations: A Case Study of Chinese-to-English Translations of Ceramic-Related Texts. Shangdong Ceramics, (04), 20-27.
Yang, F. (2023). Reflections and Inspirations from ChatGPT: A Case Study of Vietnamese Legal Translation. Chinese Science & Technology Translators Journal, (03), 27-30+4.
Yu, L. (2024). ChatGPT Lexical diversity and syntactic complexity in ChatGPT translation. Foreign Language Teaching and Research, (02), 297-307+321. doi:10.19923/j.cnki.fltr.2024.02.005.
Zhao, R., Huang, Y., Ma W., Dong, W., Xian, G., &Sun, T. (2023). Insights and Reflections of the Impact of ChatGPT on Intelligent Knowledge Services in Libraries. Journal of Library and Information Science in Agriculture, (01), 29-38.
Zhou, C., & Liu, Z. (2022). Machine Translation of Ancient Chinese Text Based on Transformer of Semantic Information Sharing. Technology Intelligence Engineering, (06), 114-127.
