Machine Translation: The Importance of Document-Level Evaluation

Research suggests that when it comes to evaluating entire documents, human translations are rated as more adequate and more fluent than machine translations. Human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. This suggests that the way machine translation is evaluated needs to evolve away from a system where machines consider each sentence in isolation.

en flag
fr flag
de flag
pt flag
es flag

Editor’s Note: Given the increased pulse rate of discussions about the use of machine translation in support of data and legal discovery tasks, the following information pieces may be beneficial for considering, contrasting, and comparing human translations and machine translations at both the sentence level and the document level.

Human Translators Are Still On Top – For Now

An extract from an article posted on MIT Technology Review

You may have missed the popping of champagne corks and the shower of ticker tape, but in recent months computational linguists have begun to claim that neural machine translation now matches the performance of human translators.

The technique of using a neural network to translate text from one language into another has improved by leaps and bounds in recent years, thanks to the ongoing breakthroughs in machine learning and artificial intelligence. So it is not really a surprise that machines have approached the performance of humans. Indeed, computational linguists have good evidence to back up this claim.

But today, Samuel Laubli at the University of Zurich and a couple of colleagues say the champagne should go back on ice. They do not dispute their colleagues’ results but say the testing protocol fails to take account of the way humans read entire documents. When this is assessed, machines lag significantly behind humans, they say.

Read the complete article at Human Translators Are Still On Top – For Now

Has Machine Translation Achieved Human Parity? A Case for Document-Level Evaluation

An abstract from a research study by Samuel Laubli, Rico Sennrich, and Martin Volk

Recent research suggests that neural machine translation achieves parity with professional human translation on the WMT Chinese–English news translation task. We empirically test this claim with alternative evaluation protocols, contrasting the evaluation of single sentences and entire documents. In a pairwise ranking experiment, human raters assessing adequacy and fluency show a stronger preference for human over machine translation when evaluating documents as compared to isolated sentences. Our findings emphasize the need to shift towards document-level evaluation as machine translation improves to the degree that errors which are hard or impossible to spot at the sentence-level become decisive in discriminating quality of different translation outputs.

Read the complete study at Has Machine Translation Achieved Human Parity? A Case for Document-Level Evaluation

In Human vs. Machine Translation, Compare Documents, Not Sentences

An extract from an article by Gino Dino

In their paper’s conclusion, Läubli, Sennrich, and Volk explain that NMT [Neural Machine Translation] is currently at a level of fluency where BLEU (bilingual evaluation understudy) scores based on a single model translation and even evaluations of non-professional human translators of sentence-level output are no longer enough.

“As machine translation quality improves, translations will become harder to discriminate in terms of quality, and it may be time to shift towards document-level evaluation, which gives raters more context to understand the original text and its translation,” the paper’s conclusion read. It further explained that document-level evaluation shows translation errors otherwise “invisible” in a sentence-level evaluation.

Read the complete article at In Human vs. Machine Translation, Compare Documents, Not Sentences

Additional Reading

Source: ComplexDiscovery

Have a Request?

If you have information or offering requests that you would like to ask us about, please let us know and we will make our response to you a priority.

ComplexDiscovery is an online publication that highlights data and legal discovery insight and intelligence ranging from original research to aggregated news for use by business, information technology, and legal professionals. The highly targeted publication seeks to increase the collective understanding of readers regarding data and legal discovery information and issues and to provide an objective resource for considering trends, technologies, and services related to electronically stored information.

ComplexDiscovery OÜ is a technology marketing firm providing strategic planning and tactical execution expertise in support of data and legal discovery organizations. Registered as a private limited company in the European Union country of Estonia, one of the most digitally advanced countries in the world, ComplexDiscovery OÜ operates virtually worldwide to deliver marketing consulting and services.

A (Brand) New Approach? Considering the Framework and Structure of eDiscovery Offerings

Today’s eDiscovery providers may benefit from the lessons learned in the creation of the Sgt. Pepper’s Lonely Hearts Club Band album by creating a concept for branding and packaging their offerings within that brand in a connected, theme-based way that represents the offerings’ promise and capability in a way that is easy to understand and remember.



Check Out the New Approach Now!

Interested in Contributing?

ComplexDiscovery combines original industry research with curated expert articles to create an informational resource that helps legal, business, and information technology professionals better understand the business and practice of data discovery and legal discovery.

All contributions are invested to support the development and distribution of ComplexDiscovery content. Contributors can make as many article contributions as they like, but will not be asked to register and pay until their contribution reaches $5.

Playing NICE? A Workforce Framework for Cybersecurity from NIST

According to Karen Wetzel, Manager of the NICE Framework, “The NICE...

New Rules? The European Regulation on Data Governance

According to the European Commission, the proposed Regulation on Data Governance...

The Price of Success? The eDiscovery Pricing Survey (Winter 2021)

Based on the complexity of data and legal discovery, it is...

Deep State? Thirteen Research Reports on the State of eDiscovery Business in 2020

As part of its coverage of the business of eDiscovery, ComplexDiscovery...

A Running List: Top 100+ eDiscovery Providers

Based on a compilation of research from analyst firms and industry...

The eDisclosure Systems Buyers Guide – 2020 Edition (Andrew Haslam)

Authored by industry expert Andrew Haslam, the eDisclosure Buyers Guide continues...

The Race to the Starting Line? Recent Secure Remote Review Announcements

Not all secure remote review offerings are equal as the apparent...

Enabling Remote eDiscovery? A Snapshot of DaaS

Desktop as a Service (DaaS) providers are becoming important contributors to...

A New Era in eDiscovery? Framing Market Growth Through the Lens of Six Eras

There are many excellent resources for considering chronological and historiographical approaches...

An eDiscovery Market Size Mashup: 2020-2025 Worldwide Software and Services Overview

While the Compound Annual Growth Rate (CAGR) for worldwide eDiscovery software...

Resetting the Baseline? eDiscovery Market Size Adjustments for 2020

An unanticipated pandemeconomic-driven retraction in eDiscovery spending during 2020 has resulted...

Home or Away? New eDiscovery Collection Market Sizing and Pricing Considerations

One of the key home (onsite) or away (remote) decisions that...

The Price of Success? The eDiscovery Pricing Survey (Winter 2021)

Based on the complexity of data and legal discovery, it is...

Deep State? Thirteen Research Reports on the State of eDiscovery Business in 2020

As part of its coverage of the business of eDiscovery, ComplexDiscovery...

A Season of Change? Eighteen Observations on eDiscovery Business Confidence in the Fall of 2020

In the fall of 2020, 77.2% of eDiscovery Business Confidence Survey...

The Continuing Case of Budgetary Constraints in the Business of eDiscovery

In the fall of 2020, 49.4% of respondents viewed budgetary constraints...

Epiq Acquires Hyperion Global Partners

According to Ziad Mantoura, SVP and General Manager for Epiq's legal...

Smarsh Acquires Digital Reasoning

According to the media release, Tim Estes, Founder and CEO of...

Reynen Court Secures Additional Funding

According to the media release, Reynen Court has secured $4.5 million...

DISCO Raises $60 Million

According to the media release, DISCO will use this investment to...

Five Great Reads on eDiscovery for November 2020

From market sizing and cyber law to industry investments and customer...

Five Great Reads on eDiscovery for October 2020

From business confidence and captive ALSPs to digital republics and mass...

Five Great Reads on eDiscovery for September 2020

From cloud forensics and cyber defense to social media and surveys,...

Five Great Reads on eDiscovery for August 2020

From predictive coding and artificial intelligence to antitrust investigations and malware,...