Artificial Intelligence and Foreign Language Learning: ChatGPT’s Limitations in Summarizing Texts
DOI:
https://doi.org/10.17507/tpls.1511.11Keywords:
artificial intelligence, teaching, foreign languages, literature, Olinda BejaAbstract
With the emergence of the widely accessible ChatGPT, teaching and assessment now face new challenges, particularly regarding the use of Artificial Intelligence (AI) tools. This paper presents a teaching experiment on the preparation of summaries of the short story “O tradutor ideal” [“The ideal translator”], by São Toméan writer Olinda Beja, based on answers provided by ChatGPT-4 and summaries produced by Chinese students in a Portuguese as a Foreign Language degree class. The methodology combines a qualitative analysis of student texts to identify AI’s influence with a longitudinal study of ChatGPT’s responses over a one-year period, aimed at observing the chatbot’s evolution. The study evaluates the potential and limitations of AI in summarizing lesser-known literary texts that are difficult to access online. Results highlight the importance of meeting challenges posed by new AI technologies without violating ethical principles. In the case of emerging literature in Portuguese, ChatGPT still shows several limitations, producing summaries that lack fidelity to the original text and literary and cultural depth. However, there has been an evolution in the responses of these Large Language Models (LLM) from April 2024 to April 2025, the period in which the study was carried out. The research highlights the need for critical reflection on the use of AI in educational and academic contexts, particularly in the teaching of literature, reinforcing the importance of adopting an ethical approach. It also emphasizes the need to closely examine the use of AI, particularly in research on Portuguese-speaking African literatures and summary writing.
References
Amaro, V. & Pires, M. J. (2024). Found in translation, lost in education: artificial intelligence’s impacts on translation tertiary education in Macao. Asian Education and Development Studies, 13(4), 269-281. https://doi.org/10.1108/AEDS-01-2024-0012
Beja, O. (2004). Pé-de-perfume [Ylang-Ylang]. Editorial Escritor.
Chaka, C. (2022). Is Education 4.0 a Sufficient Innovative, and Disruptive Educational Trend to Promote Sustainable Open Education for Higher Education Institutions? A Review of Literature Trends. Frontiers in Education, 7, 1-13. Retrieved March 10, 2025, from https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2022.824976/full
Chan, C. K.Y., & Hu, W. (2023). Students’ voices on generative AI: perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20, 1-18. https://doi.org/10.1186/s41239-023-00411-8
Darwin, D., Rusdin, D., Mukminatien, N., Suryati, N., Laksmi, E., & Marzuki, M. (2024). Critical thinking in the AI era: An exploration of EFL students’ perceptions, benefits, and limitations. Cogent Education, 11(1). https://doi.org/10.1080/2331186X.2023.2236469
Devi, M., Annamalai, M. A. R., & Veeramuthu, S. P. (2020). Literature education and industrial revolution 4.0. Universal Journal of Educational Research, 8(3), 1027-1036. https://doi.org/10.13189/ujer.2020.080337
Farhi, F., Jeljeli, R., Aburezeq, I., Dweikat, F. F., Al-shami, S. A., & Slamene, R. (2023). Analyzing the students’ views, concerns, and perceived ethics about ChatGPT usage. Computers and Education: Artificial Intelligence, 5. https://doi.org/10.1016/j.caeai.2023.100180
Ferreira, A., & Loureiro, A. (2024). As percepções dos alunos sobre a utilização da inteligência artificial nas aulas de práticas de tradução. Pratica, 7(1), 33-49. https://doi.org/10.34630/pel.v7i2.5628
Ghafar, Z., & Mohamedamin, A. (2022). Writing in English as a foreign language: How literary reading helps students improve their writing skills: A descriptive study. Canadian Journal of Educational and Social Studies, 2(6), 61-70. http://dx.doi.org/10.53103/cjess.v2i6.81
Hellström, T. (2024). AI and its consequences for the written word. Frontiers in Artificial Intelligence, 6. https://doi.org/10.3389/frai.2023.1326166
Ide, N. M., & Véronis, J. (1990). Artificial intelligence and the study of literary narrative. Poetics, 19(1-2), 37-63. https://doi.org/10.1016/0304-422X(90)90030-9
Juanda, & Afandi, I. (2024). Assessing text comprehension proficiency: Indonesian higher education students vs ChatGPT. XLinguae, 17(1), 49-68. https://doi.org/10.18355/XL.2024.17.01.04
Malaj, L. (2020). Summary strategies for literary texts in English. Studies in Logic, Grammar and Rhetoric, 65(1), 7-20. http://dx.doi.org/10.2478/slgr-2020-0043
Malmström, H., Stöhr, C., & Ou, A. W. (2023). Chatbots and other AI for learning: A survey of use and views among university students in Sweden. Chalmers Studies in Communication and Learning in Higher Education, 1, 1-16. https://doi.org/10.17196/cls.csclhe/2023/01
Marzec-Stawiarska, M. (2016). The influence of summary writing on the development of reading skills in a foreign language. System, 59(1), 90-99. https://doi.org/10.1016/j.system.2016.04.006
Matas, J. (2023). ChatGPT and the future of writing about writing. Double Helix, 11, 1-7. https://doi.org/10.37514/DBH-J.2023.11.1.09
Milano, S., McGrane, J., & Leonelli, S. (2024). Large language models challenge the future of higher education. Nature Machine Intelligence, 5, 333–334. https://doi.org/10.1038/s42256-023-00644-2
O’Halloran, K. (2024). Digital assemblages with AI for creative interpretation of short stories. Digital Scholarship in the Humanities, 39(2), 657-689. https://doi.org/10.1093/llc/fqad050
Özdemir, S. (2018). The effect of summarization strategies teaching on strategy usage and narrative text summarization success. Universal Journal of Educational Research, 6(10), 2199-2209. https://doi.org/10.13189/ujer.2018.061018
Pratama, R., & Hastutit, D. (2024). The use of artificial intelligence to improve EFL students’ writing skill. English Learning Innovation, 5(1), 13-25. https://doi.org/10.22219/englie.v5i1.30212
Rahmi, R., Amalina, Z., Andriansyah, & Rodgers, A. (2024). Does it really help? Exploring the impact of Al-Generated writing assistant on the students’ English writing. Studies in English Language and Education, 11(2), 998-1012. https://doi.org/10.24815/siele.v11i2.35875
Ramirez-Avila, M. & Barreiro, J. (2021). The effect of summarizing narrative texts to improve reading comprehension. Journal of Foreign Language Teaching and Learning, 6(2), 94-110. https://doi.org/10.18196/ftl.v6i2.11707
Russell, S. (2023). AI has much to offer humanity. It could also wreak terrible harm. It must be controlled. The Guardian, 2/4/2023. Retrieved March 18, 2025, from https://www.theguardian.com/commentisfree/2023/apr/02/ai-much-to-offer-humanity-could-wreak-terrible-harm-must-be-controlled
Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6), 1-20. https://doi.org/10.3390/healthcare11060887
Terra, E. (2023). Língua portuguesa: desenvolvendo competências de leitura e escrita [Portuguese Language: Developing reading and writing skills]. Editora Saraiva.
Tran, H. (2024). AI tools in teaching and learning English academic writing skills. Proceedings of the Asia CALL International Conference, 4, 170–187. https://doi.org/10.54855/paic.23413
Vargas-Murillo, A., Bedoya, I., & Guevara-Soto, F. (2023). The ethics of AI assisted learning: A systematic literature review on the impacts of ChatGPT usage in education. Proceedings of the 2023 8th International Conference on Distance Education and Learning, 8–13.
Vázquez-Cano, E., Ramírez-Hurtado, J. M., Sáez-López, J. M., & López-Meneses, E. (2023). ChatGPT: The brightest student in the class. Thinking Skills and Creativity, 49, 1-12. https://doi.org/10.1016/j.tsc.2023.101380
Wardle, E., & Downs, D. (2023). Writing about writing. Macmillan Learning.
Welding, L. (2023). Half of college students say using AI is cheating. BestColleges. Retrieved March 20, 2025, from https://www.bestcolleges.com/research/college-students-ai-tools-survey/
