MEANT: A highly accurate automatic metric for evaluating translation utility via semantic role labels

PhD Thesis Proposal Defence


Title: "MEANT: A highly accurate automatic metric for evaluating
translation utility via semantic role labels"

by

Miss Chi-Kiu Lo


ABSTRACT:

We propose to improve the quality of machine translation by
introducing the first SMT systems that are directly trained to
preserve meaning as defined by semantic frames.

Today's SMT systems are often able to output fluent, nearly
grammatical translations with roughly the correct words but still make
glaring errors caused by confusion of semantic roles and fail to
express meaning that is close to the input. The underlying reason is
that the development of MT systems in the past decade has been driven
by fast and cheap lexical n-gram based MT evaluation metrics which
fail to reflect translation utility. Even when human judgment clearly
indicates that one sentence translation is significantly more
meaningful, lexical similarity based evaluation metrics typically
register little difference. Semantic role labels (SRL) capture the
essential meaning of a sentence in the basic event structure - ``who
did what to whom, when, where, why and how''. As the performance of
flat n-gram oriented SMT have plateaued, we argue that it is time for
a new SRL based evaluation metric that focuses on getting the meaning
right to drive the continuous improvement of MT towards the direction
of higher utility.

In this proposal, we first introduce HMEANT, the human-involved
prototype of SRL based MT evaluation metric, that produces scores that
correlate better with human adequacy judgment than HTER, the
state-of-the-art non-automatic adequacy-oriented MT evaluation metric,
but at a lower labor cost. We then show that MEANT, the fully
automatic MT evaluation metric, correlates better with human judgment
on translation adequacy than the most commonly used automatic MT
evaluation metric. Most importantly, we present the first result of
training MT system produces MT output on MEANT that achieve improved
scores across most commonly used metrics.


Date:                   Tuesday, 4 September 2012

Time:                   2:00pm - 4:00pm

Venue:                  Room 3501
                         lifts 25/26

Committee Members:      Dr. Dekai Wu (Supervisor)
                         Prof. Fangzhen Lin (Chairperson)
 			Prof. Dit-Yan Yeung
 			Dr. Pascale Fung (ECE)


**** ALL are Welcome ****