Unveiling the New Frontier: ChatGPT-3 Powered Translation for Arabic-English Language Pairs
DOI:
https://doi.org/10.17507/tpls.1402.05Keywords:
ChatGPT-3-powered translation, machine translation, Google Translate, comparative studyAbstract
This study evaluates the aptitude of ChatGPT for Arabic-English machine translation. The main objective of this research is to scrutinize the quality of ChatGPT's translations and compare its performance against machine translation systems, such as Google Translate, which are intricately tailored for translation purposes. In addition, the study seeks to investigate the potential integration of ChatGPT into translation workflows. Furthermore, we aspire to discern whether ChatGPT's translation efficacy harmonizes with or diverges from the profound finesse exhibited by human translation expertise. To accomplish this, a comparable corpus of 1000 English sentences and their corresponding Arabic translations was employed to evaluate the translation outputs of both machine translation systems alongside a human translation reference. The corpus was sourced from Tatoeba, an open online platform and underwent electronic assessment using the BLEU (Bilingual Evaluation Understudy) metric. The results indicate a marginal advantage of ChatGPT over Google Translate in delivering high-quality translations. Upon evaluating the corpus, we ascertain that ChatGPT performs impressively well compared to specialized translation systems like Google Translate. However, despite these promising findings, it is essential to acknowledge that even the most advanced machine translation technology, ChatGPT, cannot currently match the proficiency of human translation, at least not in the near future.
References
Aiken, M. (2019). Paper An Updated Evaluation of Google Translate. Studies in Linguistics and Literature, Vol. 3, No. 3, p. 253-260.
Almansor, E.H., Al-Ani, A., Hussain, F.K. (2020). Transferring Informal Text in Arabic as Low Resource Languages: State-of-the-Art and Future Research Directions. In: Barolli, L., Hussain, F., Ikeda, M. (eds) Complex, Intelligent, and Software Intensive Systems. CISIS 2019. Advances in Intelligent Systems and Computing, vol 993. Springer, Cham. https://doi.org/10.1007/978-3-030-22354-0_17
Alves, F., Ferreira, A., & Pinto, J. (2019). The Use of Machine Translation in Small and Medium-Sized Enterprises: A case Study. Journal of Business Research, 104, p. 292-300.
Amayreh, H. & Amayreh, M. (2020). Linguistic Branching of Semantics in Arabic: a Social approach. Al-Balqa Journal for Research and Studies, 23(2), 147-156. 10.35875/1105-023-002-012
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W. et al. (2023). A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity. DOI: 10.48550/arXiv.2302.04023
Brown, T. Mann, b., Ryder, N. et al. (2020). Language Models are Few-shot Learners. Advances in Neural Information Processing Systems, 33, pp. 1877–1901.
Chen, M., Tworek, J., Jun, H. et al. (2021). Evaluating Large Language Models Trained on Code. https://doi.org/10.48550/arXiv.2107.03374.
Dai, H., Xhafa, F., Janse, B.J., LIang, H. and Ye, J. (2022). Comparative Analysis of Machine Translation and Human Translation under the Background of the Internet. International Conference on Cognitive-based Information Processing and Applications (CIPA 2021), 1(84), pp. 877-882.
Frąckiewicz, M. (2023). ChatGPT’s Contributions to Improving Translation Quality for Low-Resource Languages. Retrieved July 26, 2023 from Artificial intelligence. ChatGPT's Contributions to Improving Translation Quality for Low-Resource Languages (ts2.space).
Gao, Y., Wang, R. and Hou, F. (2023). How to Design Translation Prompts for ChatGPT: An Empirical Study. arXiv e-prints, pp.arXiv-2304. https://doi.org/10.48550/arXiv.2304.02182
Gu, W. (2023). Linguistically Informed ChatGPT Prompts to Enhance Japanese-Chinese Machine Translation: A Case Study on Attributive Clauses. https://doi.org/10.48550/arXiv.2303.15587.
Harby, A. (2023). How Accurate is Google Translate? Retrieved from: How Accurate is Google Translate? - Slator
He, J., Neubig, G. and Berg- Kirkpatrick, T. (2021). Efficient Nearest Neighbor Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5703–5714.
Jiao, W., Wang, W., Huang, J.T., Wang, X. and Tu, Z.P. (2023). Is ChatGPT a Good Translator? Yes with GPT-4 as the Engine. Retrieved June 14, 2023 from https://www.researchgate.net/publication/367359399_Is_ChatGPT_A_Good_Translator_A_Preliminary_Study.
Khoshafah, F. (2023). ChatGPT for Arabic-English Translation: Evaluating the Accuracy. DOI: 10.21203/rs.3.rs-2814154/v1
Kocmi, T. and Federmann, C. (2023). Large Language Models Are State-of-the-Art Evaluators of Translation Quality. Retrieved June 16, 2023 from 2302.14520.pdf (arxiv.org).
Koehn, P., Hoang, H., & Birch, A. (2017). The Future of Machine Translation: A Survey of Industry Experts. arXiv preprint arXiv:1707.03814.
Li, J., Wang, J., & Wang, H. (2020). The Adoption of Machine Translation by Individuals and Small-Scale Businesses: A survey. Computers in Human Behavior, 111, 106293.
Li, X. and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. In ACL 2021.
Li, Y., Yin, Y., Li, J. and Zhang, Y. (2022). Prompt-Driven Neural Machine Translation. In Findings of the Association for Computational Linguistics: ACL.
Liu, Y. Ott, M. Goyal, N. Du, J. Joshi, M. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. (2019). RoBERTa: A robustly optimized BERT pretraining approach, arXiv:1907.11692. Retrieved October 10, 2023 from doi: https://doi.org/10.48550/arXiv.1907.11692
Lu, Q., Qiu, B., Ding, L., Xie, L. and Tao, D. (2023). Error Analysis Prompting Enables Human-Like Translation Evaluation In Large Language Models: A Case Study On ChatGPT. arXiv preprint arXiv:2303.13809.
Lyu, C., Xu, J. and Wang, L. (2023). New Trends in Machine Translation Using Large Language Models: Case Examples with ChatGPT. arXiv preprint arXiv:2305.01181.
Maučec, M. and Donaj, G. (2019). Machine Translation and the Evaluation of Its Quality. Natural Language Processing - New Approaches and Recent Applications. DOI: 10.5772/intechopen.89063
Nguyen, T. T. H., Nguyen, V. H., Nguyen, T. A., & Nguyen, P. T. (2023). Using ChatGPT for Machine Translation Between Languages without Parallel Data. arXiv preprint arXiv:2302.00002.
OpenAI. (2023). Retrieved August 14, 2023 from: ChatGPT (openai.com).
Ouyang, L., Wu, J., Jiang, X. et al. (2022). Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems. Retrieved July 16, 2023 from b1efde53be364a73914f58805a001731-Paper-Conference.pdf (neurips.cc)
Peng, K., Ding, L., Zhong, Q., Shen, L., Liu, X., Zhang, M., Ouyang, Y. and Tao, D. (2023). Towards Making the Most of ChatGPT for Machine Translation. arXiv preprint arXiv:2303.13780.
Plitt, M., & O'Brien, S. (2018). The Impact of Machine Translation on the Translation Industry. Journal of Translation Studies, 21(2), 117-134.
Puchała-Ladzińska, K. (2016). Machine Translation: A Threat or an Opportunity for Human Translators?”, Studia Anglica Resoviensia T, 3, pp. 89-98, doi: 10.15584/sar.2016.13.9.
Rossi, C. & Carré, A. (2022). How to Choose a Suitable Neural Machine Translation Solution: Evaluation of MT quality. In Dorothy Kenny (ed.), Machine Translation for Everyone: Empowering Users In The Age of Artificial Intelligence, 51–79. Berlin: Language Science.
Şahin, M. and Gürses, S. (2021). English-Turkish Literary Translation Through Human-Machine Interaction. Revista Tradumàtica, 19, pp. 306-310. doi: 10.5565/rev/tradumatica.284.
Stahlberg, F. (2020). Neural Machine Translation: A review. Journal of Artificial Intelligence Research, 69(2020), pp. 343-418.
Tatoeba. (2023). Retrieved August 2, 2023 from: https://tatoeba.org/
Wang, L., Lyu, C., Ji, T., Zhang, Z., Yu, D., Shi, S., & Tu, Z. (2023). Document-Level Machine Translation With Large Language Models. arXiv preprint arXiv:2304.02210.
Warner, A. (2022). Humans Still Beat Machines When It Comes to Literary Translation. Retrieved from: MultiLingual.
Wei, J., Bosma, M., Zhao, V., Guu, K. et al. (2022). Finetuned Language Models are Zero-Shot Learners. In International Conference on Learning Representations. Retrieved July 14, 2023 from pdf (openreview.net).
World Economic Forum (WEF). (2023). Retrieved July 16, 2023 from Generative AI – A Game-Changer Society and Industry Needs to be Ready for | World Economic Forum (weforum.org).
