Measuring the Efficiency of Post-Edited Text Generated by CAT Tools: An Experimental Study
DOI:
https://doi.org/10.17507/tpls.1407.09Keywords:
post-editing, traditional human translation, translation technology, computer-assisted translation (CAT) tools, translation memoriesAbstract
Motivated by the technological advancements in computer-assisted translation (CAT) tools and the notable lack of academic research regarding their application in the Arabic-translation context, this study aims to investigate the differences in translators’ performance when comparing traditional human translation and post-edited CAT tool-generated text in terms of speed and effort. This study investigates the performance of professional translators in Saudi Arabia through traditional translation from scratch (TFS) and post-editing (PE) approaches. Data was collected from nine translators with 5–12 years of experience who had exposure to CAT tools. The participants translated an Arabic educational article into English using both methods. This study utilized Phrase CAT and Translog-II software to analyze the participants’ time and keystrokes. The results indicate that PE was significantly faster than TFS, with PE requiring 65.1% less time. PE also demanded significantly fewer keystrokes, suggesting lower technical effort. Correlations between keystrokes and time indicate a strong positive relationship in PE, implying that more technical effort correlates with increased temporal effort. These findings emphasize the efficiency of PE in enhancing productivity and suggest the importance of CAT tools and PE training for translators to meet industry demands effectively. Furthermore, this study underscores the need for continuous updates in CAT tool courses and the integration of PE training to prepare translators for constantly evolving technological landscapes.
References
Allen, J. (2001). Postediting: An integrated part of a translation software program. Language International Magazine, 13(2), 26–29.
Allen, J. (2003). Post-editing. In H. Somers (Ed.), Computers and translation: A translators guide (p. 35). John Benjamins.
Al-Rumaih, L. A. (2021). The integration of computer-aided translation tools in translator-training programs in Saudi universities: Toward a more visible state. Arab World English Journal for Translation & Literary Studies, 5(1). http://dx.doi.org/10.2139/ssrn.3802984
Aranberri, N., Labaka, G., de Ilarraza, A. D., & Sarasola, K. (2014). Comparison of post-editing productivity between professional translators and lay users. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas (pp. 20–33).
Arenas, A. G. (2019). Pre-editing and post-editing. In The Bloomsbury companion to language industry studies (p. 333). Erik Angelone, Maureen Ehrensberger-Dow and Gary Massey (eds.) Bloomsbury Academic, 2020. https://doi.org/10.5040/9781350024960.0019
Automatic Language Processing Advisory Committee. (1966). Machines: Computers in translation and linguistics (Publication 1416). Division of Behavioral Sciences, National Academy of Sciences, National Research Council.
Balling, L. W., Carl, M., & O’Brian, S. (Eds.). (2014). Post-editing of machine translation: Processes and applications. Cambridge Scholars Publishing.
Béchara, H., Orăsan, C., Parra Escartín, C., Zampieri, M., & Lowe, W. (2021, September). The role of machine translation quality estimation in the post-editing workflow. Informatics, 8(3), 61. https://doi.org/10.3390/informatics8030061
Boase-Beier, J. (2011). Stylistics and translation. In K. Malmkjaer & K. Windle, The Oxford handbook of translation studies (pp. 71–82). Oxford University Press.
Bowker, L. (2002). Computer-aided translation technology: A practical introduction. University of Ottawa Press. https://doi.org/10.1017/CBO9781107415324.004
Bowker, L. (2015). Computer-aided translation: Translator training. In S. Chan (Ed.), Routledge encyclopedia of translation technology (pp. 88–104). Routledge.
Bowker, L. (2020). Terminology management. In The Bloomsbury companion to language industry studies. London: Bloomsbury. (pp. 261–283).
Center for Research and Innovation in Translation and Translation Technology. (n.d.). Translog II. Retrieved September 22, 2022, from https://sites.google.com/site/centretranslationinnovation/translog-ii
Chan, S. (Ed.). (2014). Routledge encyclopedia of translation technology (1st ed.). Routledge.
Chesterman, A. (2004). Beyond the particular. In A. Mauranen & P. Kujamäki (Eds.), Translation universal: Do they exist? John Benjamins Publishing Company.
Christensen, T. P., & Schjoldager, A. (2010). Translation-memory (TM) research: What do we know and how do we know it? Hermes – Journal of Language and Communication in Business, 23(44), 89–101.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
Daems, J., Vandepitte, S., Hartsuiker, R., & Macken, L. (2017). Translation methods and experience: A comparative analysis of human translation and post-editing with students and professional translators. Meta: Journal des traducteurs [Meta: Translators’ Journal], 62(2), 245–270.
De Sousa, S. C., Aziz, W., & Specia, L. (2011, September). Assessing the post-editing effort for automatic and semi-automatic translations of DVD subtitles. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011 (pp. 97–103).
Densmer, L. (2014). Light and full MT post-editing explained. Rws Moravia. Retrieved June 2, 2020, from http://info.moravia.com/blog/bid/353532/Light-and-Full-MT-PostEditing-Explained Web
Do Carmo, F., & Moorkens, J. (2020). Differentiating editing, post-editing and revision. In Translation revision and post-editing (pp. 35–49). Routledge.
Doyle, M. S. (2003). Translation pedagogy and assessment: Adopting ATA’s framework for standard error marking. The ATA Chronicle, 32(11), 21–28.
Dragsted, B. (2004). Segmentation in translation and translation memory systems: An empirical investigation of cognitive segmentation and effects of integrating a TM system into the translation process. Samfundslitteratur.
Elming, J., & Balling, L. W. (2014). Investigating user behaviour in post-editing and translation using the CASMACAT workbench. In Post-editing of machine translation: Processes and applications. Cambridge Scholars Publishing: Newcastle upon Tyn (p. 147).
El-Zeini, N. T. (1994). Criteria for the evaluation of translation: A pragma-stylistic approach. Cairo University.
Espan˜a-Bonet, C., & Costa-jussa, M. R. (2016). Hybrid machine translation overview. In M. Costa-jussa, R. Rapp, P. Lambert, K. Eberle, R. Banchs, & B. Babych (Eds.), Hybrid approaches to machine translation (pp. 1–24). Springer.
Esselink, B. (2000). A practical guide to localization. John Benjamins.
European Standards for Translation Services. (2004). Translation services – Service requirements (Draft prEN15038). Retrieved May 2011 from http://web.letras.up.pt/egalvao/prEN-15038.pdf
Farahzad, F. (1992). Testing achievement in translation classes. In C. Dollerup & A. Loddegaard (Eds.), Teaching translation and interpreting: Training, talent and experience (pp. 271–278).
Fiederer, R., & O’Brien, S. (2009). Quality and machine translation: A realistic objective. The Journal of Specialised Translation, 11(11), 52–74.
García, I. (2006). Translators on translation memories: A blessing or a curse? In Translation technology and its teaching (pp. 97–105).
Garcia, I. (2009). Beyond translation memory: Computers and the professional translator. The Journal of Specialised Translation, 12(12), 199–214.
Garcia, I. (2010). Is machine translation ready yet? Target – International Journal of Translation Studies, 22(1), 7–21.
García, I. (2012). A brief history of postediting and of research on postediting. Revista Anglo Saxonica, 3(3), 291–310.
Gaspari, F., Almaghout, H., & Doherty, S. (2015). A survey of machine translation competences: Insights for translation technology educators and practitioners. Perspectives, 23(3), 333–358.
Glesne, C. & Peshkin, A. (1992). Becoming qualitative researchers: An introduction. Longman.
Gray, D. E. (2004). Doing research in the real world. SAGE Publications.
Guerberof, A. (2008). Productivity and quality in the post-editing of outputs from translation memories and machine translation. Localisation Focus – The International Journal of Localisation, 7(1), 11–21.
Guerberof, A. (2009). Productivity and quality in MT post-editing [conference paper]. Universitat Rovira I Virgilli, Spain. Retrieved July 17, 2014, from http://www.mt-archive.info/MTS-2009-Guerberof.pdf
Haji Sismat, M. A. B. (2016). Quality and productivity: A comparative analysis of human translation and post-editing with Malay learners of Arabic and English [Doctoral dissertation, University of Leeds].
Hargrave, J., & Savourel, Y. (1997). Machine assisted translation tools. Retrieved September 1, 2022, from https://patents.google.com/patent/US5724593A/en
House, J. (1997). Translation quality assessment: A model revisited. Gunter Narr.
Hu, K., & Cadwell, P. (2016). A comparative study of post-editing guidelines. Baltic Journal of Modern Computing, 4(2), 346–353.
Hutchins, J. (2000). Yehoshua Bar-Hillel (Vol. 97). John Benjamins Publishing.
Hutchins, W. J. (1986). Machine translation: past, present, future. Ellis Horwood.
Hutchins, W. J., & Somers, H. L. (1992). An introduction to machine translation (Vol. 362). Academic Press.
Karwacka, W. (2013). Constructing translation research with the use of keystroke logging—A case study. In Forum Filologiczne Ateneum (p. 93).
Kay, M. (1980). The proper place of men and machines in language translation. Machine Translation, 11(1–2), 1–20. https://doi.org/10.1023/A:1007911416676
Kiraly, D. (2014). A social constructivist approach to translator education: Empowerment from theory to practice. Routledge.
Koponen, M. (2016a). Machine translation post-editing and effort: Empirical studies on the post-editing process. A PhD thesis, University of Helsinki: Helsinki.
Koponen, M. (2016b). Is machine translation post-editing worth the effort? A survey of research into post-editing and effort. The Journal of Specialised Translation, 25, 131–148.
Koponen, M., Aziz, W., Ramos, L., Specia, L., Rautio, J., Gonzalez, M., … & Nyrkkö, S. (2012). Post-editing time as a measure of cognitive effort. In AMTA 2012 Workshop on Post-editing Technology and Practice.
Koponen, M., & Mossop, B. (2021). Translation revision and post-editing. Routledge: London. https://doi.org/10.4324/9781003096962
Krings, H. P., & Koby, G. S. (2001). Repairing texts: Empirical investigations of machine translation post-editing processes. Kent State University Press.
Läubli, S., Fishel, M., Massey, G., Ehrensberger-Dow, M., Volk, M., O’Brien, S., & Specia, L. (2013). Assessing post-editing efficiency in a realistic translation environment. In Proceedings of the 2nd Workshop on Post-editing Technology and Practice, Nice, France.
Lauffer S. (2008). The translation process: An analysis of observational methodology. Cadernos de Tradução, 9(1), 57–75. Retrieved September 1, 2022, from: http://journal.ufsc.br/.
Martínez-Mateo, R., Montero Martínez, S., & Moya Guijarro, A. J. (2017). The modular assessment pack: A new approach to translation quality assessment at the Directorate General for Translation. Perspectives, 25(1), 18–48.
McElhaney, T., & Vasconcellos, M. (1988). The translator and the postediting experience. Technology as Translation Strategy, 2, 140–148. https://doi.org/10.1075/ata.ii.28mce
Melby, A. (1979). ITS : Interactive translation system. Interactive Translation System. International Conference on Computational Linguistics, 5(1), 234–241. Retrieved September 1, 2022, from file:///Users/SK/Documents/MyResearch/Library/articles/C80-1064.pdf
Melby, A. (1981). Translators and machines – Can they cooperate? Meta: Journal Des Traducteurs [Meta: Translators’ Journal], 26(1), 23. https://doi.org/10.7202/003619ar
Melby, A. (1982). Multi-level translation aids in a distributed system. In Proceedings of the Ninth International Conference on Computational Linguistics (pp. 215–220). Retrieved September 1, 2022, from http://ww2.cs.mu.oz.au/acl/C/C82/C82-1034.pdf
Melby, A. (1992). The translator workstation. In John Newton (Ed.), Computers in translation: A practical appraisal (pp. 147–165). https://doi.org/10.1075/ata.vii.14me
Melby, A. K. (1983, February). Computer-assisted translation systems: The standard design and a multi-level design. In First Conference on Applied Natural Language Processing, pages 174–177, Santa Monica, California, USA. Association for Computational Linguistics.
Moorkens, J., Toral, A., Castilho, S., & Way, A. (2018). Translators’ perceptions of literary post-editing using statistical and neural machine translation. Translation Spaces, 7(2), 240–262.
Munday, J. (2016). Introducing translation studies: Theories and applications. Routledge.
Newmark, P. (1995). A textbook of translation. Longman.
Nida, E. A. (1964). Toward a science of translating: With special reference to principles and procedures involved in Bible translating. Brill Archive.
Nida, E. A. (2001, November). Language and culture-contexts in translating. Shanghai Foreign Language Education Press.
Nord, C. (2001). Translating as a purposeful activity-functionalist approaches explained. Shanghai Foreign Language Education Press.
O’Brien, S. (2007). An empirical investigation of temporal and technical post-editing effort. Translation and Interpreting Studies – The Journal of the American Translation and Interpreting Studies Association, 2(1), 83–136.
O’Brien, S. (2010). Introduction to post-editing: Who, what, how and where to next? In Proceedings of the 9th Conference of the Association for Machine Translation in the Americas: Tutorials.
O’Brien, S. (2011). Towards predicting post-editing productivity. Machine Translation, 25(3), 197–215.
Phelan, M. (2017). Analytical assessment of legal translation: A case study using the American Translators Association framework. The Journal of Specialised Translation, 27, 89–210.
Plitt, M., & Masselot, F. (2010). A productivity test of statistical machine translation post-editing in a typical localisation context. The Prague Bulletin of Mathematical Linguistics, 93, 7–16.
Samman, H. M. (2022). Evaluating machine translation post-editing training in undergraduate translation programs—An exploratory study in Saudi Arabia [Doctoral dissertation, University of Southampton].
Schäfer, F. (2003). MT post-editing: How to shed light on the “unknown task”. Experiences at SAP. In EAMT Workshop: Improving MT through other language technology tools: Resources and tools for building MT (p. 3).
Secară, A. (2005). Translation evaluation: A state of the art survey. In Proceedings of the eCoLoRe/MeLLANGE Workshop (pp. 39–44). St. Jerome Publishing.
Shakir, R. M. F., & Al-Ali, K. K. (2021). Post-editing as a creative tool in improving the quality of the product of translation students. Adab Al-Basrah, 97(1), 1227-1235.
Sharma, V. (2015). The relevance of addition, omission and deletion (AOD) in translation. International Journal of Translation, 27(1–2), 1–12.
Somers, H. (Ed.). (2003). Computers and translation: A translator’s guide. John Benjamins.
Tatsumi, M. (2010). Post-editing machine translated text in a commercial setting: Observation and statistical analysis [Doctoral dissertation, Dublin City University].
Temnikova, I. (2010, May). Cognitive evaluation approach for a controlled language post-editing experiment. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10).
Terninko, J. (2018, January). 2018 European language industry survey. European Language Industry Association. Retrieved September 1, 2022, from https://elia-association.org/2018/01/2018-european-language-industry-survey/
Tetnowski, J. (2015). Qualitative case study research design. Perspectives on Fluency and Fluency Disorders, 25(1), 39–45.
Vazquez, L. M., Vazquez, S. R., & Bouillon, P. (2013). Comparing forum data post-editing performance using translation memory and machine translation output: A pilot study. In Proceedings of Machine Translation Summit XIV (pp. 249–256).
