A model for translation accuracy evaluation and measurement: a quantitative approach

Translation quality assessment (TQA) has been part of the translating process since Marcus Tullius Cicero (106-43BCE), and earnest studies on TQA have been conducted for several decades, but there has been no breakthrough in standardized TQA. Though the importance of TQA has been stressed, agreement on specific means of TQA has not been reached. As Chesterman and Wagner summarize, "Central to translation [...]," "[q]uality assessment is so complicated - especially if it is to be objective and reproducible" (2002: 80-81). The approaches to TQA published throughout the past millennia, by and large, are qualitative. "Whereas there is general agreement on the requirement for a translation to be 'good,' 'satisfactory,' or 'acceptable,' the definition of acceptability and of the means of determining it are matters of ongoing debate and there is precious little agreement on specifics" (Williams, 2004: xiv). Most published TQA approaches are neither objective nor reproducible. -- My study proposes a model for fuzzy standardized TQA through a quantitative approach, which expresses TQA results in numerical terms in a consistent manner. My model is statistics-based, practice-based and practice-oriented. It has been independently tested by eleven professors from four countries, fifteen senior United Nations translators, and fifty reader evaluators. My contrastive analysis of 23,000 pages of bilingual and multilingual texts has identified the unit of translation - the orthographic sentence in context, which is also verified by the results of an international survey among 66 professional translators, the majority of whom also confirm that they evaluate translations sentence by sentence in context. Halliday and Matthiessen's functional grammar theory, among others, provides my model for quantitative TQA with its theoretical basis, while the international survey, the necessary data. My model proposes a set of six Fuzzy Functional Translation Grammar terms, a grammar concept general enough to cover all grammar units in the translated orthographic sentence. Each term represents one type of error which contains from one to three sub-categories. Each error is assigned a value - the mean of the professional markers' deductions for relevant artificial errors and original errors. A marking scheme with sixteen variables under eight attributes is thus created. Ten marks are assigned to each unit of TQA, the sentence. For easy calculation, an arithmetic formula popularly used in statistics (Ex/n ) is adopted. With the assistance of a simple calculator, the evaluator can calculate the grade of a sentence, a sentence group, and the overall grade for an entire TT, regardless of its length. -- Perfect reliability or validity in any form of measurement is unattainable. There will always be some random error or noise in the data (McClendon, 2004: 7). Since it is the first of its type, I do not claim that my model is perfect. Variation has been found in the results of the testing performed by scholars and professional translators, but further testing based on two "easy" (markers' comment) sentences by the 50 reader evaluators respectively achieves 98% and 100% consistency, which indicates that markers' competence may equal constancy or that proper marker training and/or strict marker examination will minimize inconsistency among professional markers. My model, whose formulas withstand testing at the theoretical level and in practice, is not only ready for application, but it has profound implications beyond TQA, such as use in machine translation, and for other subjects like the role of the sentence in translation studies and translating practice.