PORT : a precision-order-recall MT evaluation metric for tuning

  1. (PDF, 287 KB)
AuthorSearch for: ; Search for: ; Search for:
Proceedings titleProceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012)
ConferenceThe 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), 8-14 July 2012, Jeju Island, Korea
Pages930939; # of pages: 10
AbstractMany machine translation (MT) evaluation metrics have been shown to correlate better with human judgment than BLEU. In principle, tuning on these metrics should yield better systems than tuning on BLEU. However, due to issues such as speed, requirements for linguistic resources, and optimization difficulty, they have not been widely adopted for tuning. This paper presents PORT1 , a new MT evaluation metric which combines precision, recall and an ordering metric and which is primarily designed for tuning MT systems. PORT does not require external resources and is quick to compute. It has a better correlation with human judgment than BLEU. We compare PORT-tuned MT systems to BLEU-tuned baselines in five experimental conditions involving four language pairs. PORT tuning achieves consistently better performance than BLEU tuning, according to four automated metrics (including BLEU) and to human evaluation: in comparisons of outputs from 300 source sentences, human judges preferred the PORT-tuned output 45.3% of the time (vs. 32.7% BLEU tuning preferences and 22.0% ties).
Publication date
PublisherCurran Associates
AffiliationInformation and Communication Technologies; National Research Council Canada
Peer reviewedYes
NPARC number21267975
Export citationExport as RIS
Report a correctionReport a correction
Record identifier376a485b-e4c4-4ffe-bc8a-cfae93085e10
Record created2013-03-27
Record modified2016-05-09
Bookmark and share
  • Share this page with Facebook (Opens in a new window)
  • Share this page with Twitter (Opens in a new window)
  • Share this page with Google+ (Opens in a new window)
  • Share this page with Delicious (Opens in a new window)