View source for Machine translation software usability

The sections below give objective criteria for evaluating the usability of [[Machine translation|machine translation]] software output.
== Stationarity or canonical form ==
{{main|Round-trip translation}}
Do repeated translations converge on a single expression in both languages?  I.e. does the translation method show [[Stationary process|stationarity]] or produce a [[Canonical form|canonical form]]? Does the translation become stationary without losing the original meaning?  This metric has been criticized as not being well correlated with [[BLEU]] (BiLingual Evaluation Understudy) scores.<ref name="somers05">{{cite journal|last1=Somers|first1=Harold|title=Round-trip translation: What is it good for?|journal=Proceedings of Australasian Language Technology Workshop ALTW 2005|date=2005|pages=127–133|url=http://personalpages.manchester.ac.uk/staff/harold.somers/RoundTrip.doc|location=Sydney}}</ref>

== Adaptive to colloquialism, argot or slang ==

Is the system adaptive to [[Social:Colloquialism|colloquialism]], [[Social:Argot|argot]] or [[Social:Slang|slang]]? The [[Social:French language|French language]] has many rules for creating words in the speech and writing of [[Social:Popular culture|popular culture]].  Two such rules are:  (a) The reverse spelling of words such as ''femme'' to ''meuf''.  (This is called verlan.) (b) The attachment of the suffix ''-ard'' to a noun or verb to form a proper noun.  For example, the noun ''[[Social:Faluche|faluche]]'' means "student hat".  The word ''faluchard'' formed from ''faluche'' colloquially can mean, depending on context, "a group of students", "a gathering of students" and "behavior typical of a student".  The Google translator as of 28 December 2006 doesn't derive the constructed words as for example from rule (b), as shown here:

<blockquote>
Il y a une chorale falucharde mercredi, venez nombreux, les faluchards chantent des paillardes! ==> ''There is a choral society falucharde Wednesday, come many, the faluchards sing loose-living women!''
</blockquote>

French argot has three levels of usage:<ref>[http://chitlinsandcamembert.blogspot.com/2005/10/agony-of-argot.html "The Agony of Argot", Chitlins & Camembert, October 28, 2005]</ref>
#''familier'' or friendly, acceptable among friends, family and peers but not at work
#''grossier'' or swear words, acceptable among friends and peers but not at work or in family
#''verlan'' or ghetto slang, acceptable among lower classes but not among middle or upper classes

The United States [[Organization:National Institute of Standards and Technology|National Institute of Standards and Technology]] conducts annual evaluations [https://www.nist.gov/speech/tests/mt/] of [[Machine translation|machine translation]] systems based on the [[BLEU]]-4 criterion [https://www.nist.gov/speech/tests/mt/doc/mt06_evalplan.v4.pdf].  A combined method called IQmt which incorporates BLEU and additional metrics NIST, GTM, ROUGE and METEOR has been implemented by Gimenez and Amigo [http://www.lsi.upc.edu/~nlp/IQMT/IQMT.v1.0.pdf].

== Well-formed output ==

Is the output grammatical or well-formed in the target language? Using an interlingua should be helpful in this regard, because with a fixed interlingua one should be able to write a grammatical mapping to the target language from the interlingua. Consider the following [[Social:Arabic language|Arabic language]] input and [[Social:English language|English language]] translation result from the Google translator as of 27 December 2006 [http://www.google.com/language_tools?hl=en]. This Google translator output doesn't parse using a reasonable English grammar: 	 
 		 
 	<blockquote> 	 
 	وعن حوادث التدافع عند شعيرة رمي الجمرات -التي كثيرا ما يسقط فيها العديد من الضحايا- أشار الأمير نايف إلى إدخال "تحسينات كثيرة في جسر الجمرات ستمنع بإذن الله حدوث أي تزاحم". 	 
	==> 	 
	And incidents at the push Carbuncles-throwing ritual, which often fall where many of the victims - Prince Nayef pointed to the introduction of "many improvements in bridge Carbuncles God would stop the occurrence of any competing." 	 
	</blockquote>

== Semantics preservation ==

Do repeated re-translations preserve the [[Philosophy:Semantics|semantics]] of the original sentence? For example, consider the following English input passed multiple times into and out of French using the Google translator as of 27 December 2006: 	 
 		 
 	<blockquote> 	 
	Better a day earlier than a day late. ==> 	 
 		 
	''Améliorer un jour plus tôt qu'un jour tard.'' ==> 	 
		 
 	To improve one day earlier than a day late. ==> 	 
		 
	''Pour améliorer un jour plus tôt qu'un jour tard.'' ==> 	 
		 
	To improve one day earlier than a day late. 	 
	</blockquote>

As noted above and in,<ref name="somers05" /> this kind of round-trip translation is a very unreliable method of evaluation.

== Trustworthiness and security ==

An interesting peculiarity of [http://www.google.com/translate_t?langpair=en|es Google Translate] as of 24 January 2008 (corrected as of 25 January 2008) is the following result when translating from English to Spanish, which shows an embedded joke in the English-Spanish dictionary which has some added poignancy given recent events:

 	<blockquote> 	 
	Heath Ledger is dead ==> 	 
 		 
	''Tom Cruise está muerto'' 
	</blockquote> 	
		  	
This raises the issue of trustworthiness when relying on a machine translation system embedded in a Life-critical system in which the translation system has input to a Safety Critical Decision Making process.  Conjointly it raises the issue of whether in a given use the software of the machine translation system is [[Security|safe]] from hackers.

It is not known whether this feature of Google Translate was the result of a joke/hack or perhaps an unintended consequence of the use of a method such as [[Statistical machine translation|statistical machine translation]].  Reporters from CNET Networks asked Google for an explanation on January 24, 2008; Google said only that it was an "internal issue with Google Translate".<ref>[http://www.news.com/8301-13577_3-9857280-36.html?tag=newsmap "Google Translate bug mixes up Heath Ledger, Tom Cruise", by Caroline McCarthy], CNET Networks, January 24, 2008</ref>  The mistranslation was the subject of much hilarity and speculation on the Internet.<ref>[http://gawker.com/5002510/tom-cruise-is-spanish-for-heath-ledger '"Tom Cruise" is Spanish for "Heath Ledger"', gawker.com, January 24, 2008] {{webarchive|url=https://web.archive.org/web/20080128172538/http://gawker.com/5002510/tom-cruise-is-spanish-for-heath-ledger |date=January 28, 2008 }}</ref><ref>[http://rayhey2.blogspot.com/2008/01/tom-cruise-est-muerto.html "Tom Cruise está muerto", Ray Leon Blog Project, January 24, 2008] {{webarchive|url=https://web.archive.org/web/20081029004505/http://rayhey2.blogspot.com/2008/01/tom-cruise-est-muerto.html |date=October 29, 2008 }}</ref>

If it is an unintended consequence of the use of a method such as [[Statistical machine translation|statistical machine translation]], and not a joke/hack, then this event is a demonstration of a potential source of critical unreliability in the statistical machine translation method.

In human translations, in particular on the part of interpreters, selectivity on the part of the translator in performing a translation is often commented on when one of the two parties being served by the interpreter knows both languages.

This leads to the issue of whether a particular translation could be considered ''verifiable''.  In this case, a converging round-trip translation would be a kind of verification.

==See also==
*[[Software:Comparison of machine translation applications|Comparison of machine translation applications]]
*[[Evaluation of machine translation]]
*[[Round-trip translation]]
*[[Philosophy:Translation|Translation]]

==Notes==
{{reflist}}

==References==

* Gimenez, Jesus and Enrique Amigo.  (2005) [http://www.lsi.upc.edu/~nlp/IQMT/IQMT.v1.0.pdf IQmt: A framework for machine translation evaluation].
* NIST.  [https://www.nist.gov/speech/tests/mt Annual machine translation system evaluations] and [https://www.nist.gov/speech/tests/mt/doc/mt06_evalplan.v4.pdf evaluation plan].
* Papineni, Kishore, Salim Roukos, Todd Ward and Wei-Jing Zhu. (2002) BLEU: A Method for automatic evaluation of machine translation.  Proc. 40th Annual Meeting of the ACL, July, 2002, pp.&nbsp;311–318.

[[Category:Applications of artificial intelligence]]
[[Category:Computational linguistics]]
[[Category:Machine translation software|*]]
[[Category:Natural language processing]]

{{Sourceattribution|Machine translation software usability}}