Machine Translation: Ingredients for Productive and Stable MT deployments - Part 1

By Laurie Gerber,
treasurer, Association for Machine Translation in the Americas president,
International Association for Machine Translation

This is the first of three planned articles reporting on Machine Translation industry developments that emerged at the AMTA (Association for Machine Translation in the Americas) conference held in Honolulu, Hawaii, October 21-25, 2008. AMTA conferences are held every other year, and have the goal of bringing together users, developers and researchers on MT. This format provides a unique opportunity to catch up on everything MT - from long term research developments to current deployments and products.

This first article reports on a workshop in which current users of MT shared and discussed the components of successful MT deployments. Future articles will address what is happening in MT research, and applications of machine translation in the US Government which may be less familiar to readers of Client Side News.

Before I dive in, I want to introduce a couple of terms I'll use in all three articles to summarize the two main types of MT use and their common characteristics. Translation for information assimilation and translation for dissemination.

Translation for dissemination will be the most familiar type to this audience: Organizations that use MT to disseminate information typically value translation quality over speed. They usually control the source text and may have authored it with a view to eventual translation. The text they need to translate will focus on a single topic, or perhaps on the general product lines and operations of a single company or organization (as narrow as HP all-in-one printers, or as broad as European parliamentary proceedings.) Information is being disseminated from one language into the many languages of the clients for that information. Dissemination users often have the resources (both financial and legacy of text and dictionaries) for customizing MT. The most typical example is product documentation.

Organizations that use MT for assimilation typically value speed and volume over polished quality. They are also typically translating many foreign languages into their own language, and they do not have any control over the source text. Operations trying to assimilate information may use MT together with other analytic tools to identify proper names, or other important nuggets of information. They may use translation as a way to identify whole texts, or just passages, for an authoritative human translation. The automatically translated text may or may not be used as the basis for such human translations. Users of translation for assimilation may specialize in particular topic areas, but often they need general purpose translation capability that can handle any topic. Such users rarely have the resources to do extensive subject area customization of MT engines.

Recipes for success: Ingredients of stable and productive MT deployments

A group of 35 current and prospective users of machine translation gathered for a full-day workshop on October 21, the day before the main AMTA conference. 12 presenters shared accounts of their work with machine translation in applications that included information assimilation operations for U.S. military intelligence and health incident detection, to information dissemination operations on health and agriculture at the Pan American Health Organization, and self-serve technical support information at Intel Corporation. In addition to the 12 presenters, many of the audience members also had relevant experience that they shared during discussion periods. I will mention a few of the insights. But a more complete report on all of the presentations and workshop outcomes will appear early in 2009 on the AMTA website (www.amtaweb.org). The presentations were divided into several topic areas. I will touch on two: Project Justification and Evaluation.
Project justification

Four presenters explicitly addressed this topic - two from the U.S. Government, one from an NGO (non-governmental organization), and one from industry. These focused presentations emphasized project justification for new projects. But many of the presentations touched on how their organization views the ongoing ROI (return on investment) for their project.

Careful planning versus crisis response: When launching technology projects, two of the presenters (one in the U.S. Government and one in industry - Intel Corporation) described project justification and startup processes that involve systematically building a case for introducing the new technology, demonstrating likely benefit, conducting research into technology options, and gathering requirements and support within the organization. A recent project at Intel has yielded a stable, well-supported, small-scale deployment that is demonstrating its usefulness on a well-focused problem, and is in a good position to be extended in a transparent way to additional languages and text types. The commercial project at Intel Corporation involves fully automated translation of technical knowledgebase articles into Spanish to provide self-serve support. The technical articles now being automatically translated were historically available only in English. The Intel team had to demonstrate strong ROI for a 2-5 year time horizon in both cost savings and reduction in the number of support calls. The actual results are quite impressive - adoption of English->Spanish MT for certain technical support articles has succeeded in deflecting support calls, and the quality is high enough that some human translation efforts have discontinued. The group that conducted this pilot project is now preparing to expand the project to additional languages and text types.

The other two presenters on this topic (one in the US Government and the other in an NGO) described highly innovative projects where the project definition emerged from visionary thinking, rather than from requirements gathering. In each case, the proposed system had a modest but promising prototype when a crisis occurred for which the system offered a unique and powerful solution, and support flowed from resources allocated to address the crisis. The NGO project (the GHPHIN system at Public Health Canada) sought to identify early indicators of public health problems in online news sources in 7 languages. The crisis that accelerated the project was the outbreak of SARS in late 2002, which the system correctly detected months before the World Health Organization identified it by conventional methods. The U.S. Government project provided intelligence analysts a system for transcribing and translating foreign news broadcasts. The system helps intelligence analysts locate valuable information in foreign language news broadcasts without having to watch every minute. The system offers the further value of supplying pictures and video clips that add value to analysts' reports. The crisis that speeded the way for deployment of the video monitoring system was the post 9/11 need to monitor Arabic language television broadcasts. In both cases, the usual justification process was cut short when the prototype was able to solve the emergent problem.

Other presenters touched on their ongoing justification to management. PAHO (the Pan American Health Organization) which translates health and agricultural documents between English, Spanish and Portuguese utilizes cost savings and productivity justification which they monitor with an automated workflow and project management system that tracks translator productivity and costs. PAHO's story was unique among the presenters because the MT systems they use were all developed in-house beginning in the 1970s. A small development and maintenance team has been continually active since its inception. ROI at PAHO demonstrates the value of the system costs which include that development and maintenance effort.

Evaluation

A participant working on another part of the Intel project reported on the pre-deployment evaluation of translation quality. Typically, organizations bring in their translators to evaluate MT quality - who else understands translation as well? But translators also tend to judge MT very harshly. This is often seen as self-preservation, and there is certainly an element of that. But professional translators are trained to adhere to a very high standard of accuracy. And professional ethics prevent them from accepting any translation job that they cannot deliver at the highest levels of quality. Is it any wonder that they find MT offensive?

At the same time, sometimes the end user's standard isn't "the highest level of quality". In the case in point, Intel's standard was the ability to deflect support calls for a language that had very little technical support content before the project began. When the company was evaluating machine translation output to assess feasibility of an MT solution, human linguists rated the MT generally inadequate. But the company's representatives in central and south America evaluated the MT output as quite adequate for the purpose at hand - to provide better support to a Spanish-speaking audience, and reduce the number of support calls that resulted from the lack of Spanish language self-help content.

When the system was deployed, user responses to the question, "did this information help answer your question" actually exceeded the satisfaction levels projected even by the regional representatives. English->Spanish automatically translated content was somewhat behind the satisfaction rates for English speaking users working with the English authored content (about 43% satisfaction for the Spanish machine-translated articles vs 53% satisfaction for English originals) But user satisfaction on the Spanish machine-translated articles was well ahead of the human translated content for French, German, Italian and Turkish, with user satisfaction rates between 34%-40%.

Next Time

The group that gathered for the first Recipes for Success workshop expressed their interest in future workshops, to continue to explore the ingredients of productive MT use. If you would like to receive updates on future workshops, please send an email to: gerbl@pacbell.net

Author Bio

Laurie Gerber has worked in the field of machine translation for over 20 years, including system development, research, and business development. Throughout this time, users and usability have been a defining interest. Laurie became an independent consultant in April 2008 in order to help user organizations create successes with machine translation and other language technology.

Laurie has been active in the machine translation professional community since 1992 and is currently treasurer of the Association for Machine Translation in the Americas, and President of the International Association for Machine Translation. http://www.amtaweb.org




Published - January 2009



ClientSide News Magazine - www.clientsidenews.com

Corporate Blog of Elite - Professional Translation Services serving ASEAN & East Asia