Data-driven Asian Adapted MQM Typology and Automation in Translation Quality Workflows


Inter-annotator agreement
multidimensional quality metrics
translation quality workflows
annotation procedures of quality processes
East Asian languages
quality estimation

How to Cite

Silva, B. ., Buchicchio, M., van Stigt, D., Stewart, C., Moniz, H., & Lavie, A. (2024). Data-driven Asian Adapted MQM Typology and Automation in Translation Quality Workflows. The Journal of Specialised Translation, (41), 98–126.


In this study we aim to test the impact of applying translation error taxonomies oriented towards European Languages in the annotation of Asian Languages. We aim to demonstrate how an error typology adapted for the latter languages can not only result in more linguistically accurate annotations, but also how this can be applied to automating and scaling translation quality evaluation.

As such, we propose a Translation Errors Typology that aims to cover the shortcomings of the Multidimensional Quality Metrics (Lommel et al. 2014) framework (MQM) in what concerns the annotation of the East Asian Languages of Mandarin, Japanese and Korean. The effectiveness of the typology here proposed was tested by analysing the Inter-annotator agreement (IAA) scores obtained, in contrast with the typology proposed by Ye and Toral (2020) and the Unbabel Error Typology1. Finally, we propose a way of automating Translation Quality Workflows through a Quality Estimation (QE) technology that is able to predict the MQM scores of machine translation outputs at scale with a fair correlation with the human judgement produced by applying the East Asian Languages MQM module proposed in this study.
Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Copyright (c) 2024 Beatriz Silva, Marianna Buchicchio, Daan van Stigt, Craig Stewart, Helena Moniz, Alon Lavie