Machine translation (MT) has advanced significantly in recent years. Neural machine translation, customized MT engines, and AI-powered translation tools can quickly process large volumes of multilingual content. But speed is only part of the picture, as quality also matters. For machine translation evaluation, the BLEU score is one of the most established methods for measuring quality.
Researchers have used BLEU as an MT benchmark for more than two decades. Even with newer evaluation methods available, BLEU remains useful for comparing machine translation systems and tracking performance over time.
The number itself, however, only tells part of the story. A BLEU score of 70 does not mean a translation is 70% accurate, and scores produced under different testing conditions cannot always be compared directly.
Knowing how BLEU works and what the results mean can help put the scores in context.
What is a BLEU score?
BLEU stands for Bilingual Evaluation Understudy. IBM researchers introduced the metric in 2002 as an automated metric for evaluating machine translated text.
BLEU compares a machine translation with one or more human reference translations. It then measures how closely the words and phrases match.
BLEU looks for matching words and sequences of words, known as n-grams, between the machine translation and the reference translation.
Consider a human reference translation:
The patient should take this medication once daily.
An MT system produces:
The patient should take this medication once daily.
The significant overlap between the two versions would contribute to a high BLEU score.
Now consider another translation:
The patient should take this medicine one time per day.
The meaning is similar, but the wording is different. BLEU looks for matching words and phrases. Because of this, the second translation may get a lower score even if both versions are accurate.
This distinction is important: BLEU measures similarity to reference translations. It does not measure translation accuracy as a percentage.
How are BLEU scores calculated?
BLEU goes beyond counting individual matching words. It evaluates sequences of different lengths to determine how closely machine-translated content resembles the reference translations.
A standard BLEU evaluation typically considers:
- 1-grams: individual words
- 2-grams: sequences of two words
- 3-grams: sequences of three words
- 4-grams: sequences of four words
As more words and phrases align with the reference translations, the BLEU score generally increases.
BLEU also uses modified precision, which limits how often a matching word can contribute to the score. This prevents an MT system from receiving an artificially high result by repeatedly producing words found in the reference.
Another component, the brevity penalty, accounts for translations substantially shorter than the reference. Without it, an MT system could produce a short translation with several matching phrases but leave out important information.
BLEU calculates a score between 0 and 1, but most evaluations report results on a scale from 0 to 100. A BLEU value of 0.65, for example, would typically appear as a BLEU score of 65.
Does a BLEU score of 70 mean 70% accuracy?
No. This is one of the most important distinctions to understand when interpreting BLEU results.
A BLEU score of 70 does not mean:
- 70% of the translation is accurate
- 70% of the words are correct
- 70% of translations would pass human review
Instead, the score shows how closely the machine translation matches the reference translations.
A higher BLEU score generally means the MT output more closely resembles the reference translations used in the evaluation. This makes the score a useful benchmark, but not a percentage of translation accuracy.
What is a “good” BLEU score?
Higher BLEU scores generally indicate greater similarity between machine-generated output and human reference translations. However, there is no universal BLEU score defining a good translation.
You may see general BLEU ranges used to describe results as low, moderate, or high. These ranges can provide context, but they do not define universal MT quality thresholds.
Several factors can influence the result, including:
Language pair: Languages differ significantly in grammar, morphology, sentence structure, and word order.
Content type: Technical content may score differently than everyday language. Some types of content allow for more ways to say the same thing.
Reference translations: The number, quality, and wording of the reference translations can influence the score.
Test content: Scores generated from different evaluation sets are not necessarily comparable, even when testing the same language pair.
Scoring methodology: Tokenization, capitalization, and other technical choices can affect BLEU results.
For these reasons, a BLEU score means more when you compare it with other results from the same test.
How should you compare BLEU scores?
BLEU becomes particularly useful when comparing MT engines under consistent testing conditions.
For example, an organization may start with a general MT engine. It can then train the engine to better match its language and content.
Consider this example:
Baseline MT engine: BLEU 42
Trained MT engine: BLEU 58
The 16-point increase shows the trained engine produced translations closer to the reference translations.
It does not mean translation accuracy improved by 16%.
This distinction makes BLEU useful for measuring the impact of MT engine training and customization. Organizations can evaluate how performance changes after incorporating industry terminology, preferred language, domain-specific content, or other relevant training data.
For the comparison to be meaningful, the conditions should remain consistent. Both engines should be evaluated using the same source content, reference translations, and scoring methodology.
In this type of controlled evaluation, the change in BLEU score can often provide more useful information than the absolute number alone.
Why can BLEU scores vary by language?
BLEU scores should generally be interpreted within the context of each language pair rather than used to create a universal ranking across languages.
Suppose one trained engine receives a BLEU score of 68, while another language receives a score of 55. The difference does not necessarily mean the first engine produces better translations.
Language structure can have a significant impact on BLEU. Differences in morphology, grammar, word order, and tokenization can affect how closely a valid translation matches a reference.
The test content and reference translations can also differ between languages.
When evaluating MT performance across several languages, comparing each trained or customized engine with its corresponding baseline can provide more meaningful context than comparing absolute scores across unrelated language pairs.
How can MT engine training and customization affect BLEU scores?
General-purpose MT engines are designed to handle a broad range of subjects and content. Organizations often have much more specific language requirements.
A healthcare organization, for example, may regularly use terminology associated with medications, appointments, procedures, and patient communications. A health plan may need language related to claims, benefits, coverage, enrollment, and member communications. Other organizations may have product names, technical terminology, or preferred translations unique to their business.
Training or customizing an MT engine around relevant terminology and content can influence how closely its output aligns with the language an organization expects.
BLEU provides one way to measure the impact of those changes.
A team can establish a baseline, train or customize the engine, and then evaluate both versions using consistent test content. The comparison provides a repeatable benchmark for determining whether the changes are moving MT output closer to the desired reference translations.
The same approach can be used as an engine continues to change. Consistent testing can help teams evaluate new model versions and monitor performance over time.
What are the limitations of BLEU?
BLEU provides a scalable and repeatable benchmark, but it cannot fully determine whether a translation communicates the intended meaning.
One limitation comes from the number of valid ways to translate the same sentence.
Consider:
Contact your health plan if you have questions about your coverage.
Another translator might write:
If you have questions about your coverage, contact your health plan.
The wording and word order differ, but the meaning remains essentially the same. BLEU may give one version a higher score simply because its words and phrases are closer to the reference.
The opposite problem can also occur.
Consider:
Take one tablet once daily.
compared with:
Take one tablet twice daily.
Most of the words match, but one word significantly changes the medical instruction.
Or consider:
This service is covered by your plan.
compared with:
This service is not covered by your plan.
The sentences have substantial lexical overlap, but their meanings are opposite.
BLEU does not inherently understand the significance of an individual word or error. It measures patterns of similarity rather than the meaning, context, or potential impact of a translation.
For specialized content, particularly in areas such as healthcare, insurance, and legal services, these distinctions can be important.
Why does consistent testing matter?
The usefulness of a BLEU score depends heavily on how the evaluation is conducted.
Ideally, MT systems are evaluated using a held-out test set, or content separate from the material used to train or customize the engine. This approach provides a better indication of how an engine may perform when presented with new content.
Test-set size matters, too. BLEU was primarily designed as a corpus-level metric, meaning it is more informative when evaluating a collection of translations rather than assigning a quality score at a sentence level.
Consistency in the scoring methodology is also important. Technical decisions, including how text is divided into tokens before comparison, can influence BLEU results.
Standardized implementations such as SacreBLEU were developed to make BLEU evaluations easier to reproduce by documenting important elements of the scoring configuration.
When the same evaluation content and methodology are used consistently, BLEU can provide a useful benchmark for identifying changes in MT performance.
Is BLEU still relevant for modern AI translation?
BLEU was introduced in 2002, well before today’s neural MT systems and large language models. Translation technology has changed considerably since then, and evaluation methods have evolved alongside it.
BLEU remains widely used because it is fast, repeatable, scalable, and familiar across the machine translation industry.
Newer metrics can provide additional perspectives. chrF, for example, evaluates character-level similarities, while COMET uses neural models to assess translations with greater consideration for semantic meaning.
These newer approaches do not necessarily make BLEU irrelevant. Instead, they highlight why translation quality is difficult to represent with a single number.
Different evaluation methods answer different questions. BLEU can measure similarity to reference translations at scale. Semantic metrics can provide additional insight into meaning. Professional linguists can evaluate context, terminology, fluency, and the significance of individual errors.
Looking beyond a single translation quality score
BLEU provides a valuable way to benchmark MT performance, particularly when comparing engines under consistent testing conditions. It can help organizations measure the impact of engine training, evaluate changes over time, and better understand how closely MT output aligns with desired translations.
Translation quality, however, involves more than similarity to a reference.
At Propio, we use a range of language technology, quality of machine translation measurement methods, and linguistic expertise to support different content and use cases. This can include machine translation, AI-powered solutions, automated quality measurement, customized technology, human evaluation, and human review. The right combination depends on factors such as the language, content, audience, volume, and level of risk involved.
BLEU can provide one valuable data point within this broader quality process. Human expertise and additional evaluation methods provide context where an automated score alone may not tell the full story.
For a closer look at the different factors involved in evaluating translated content, read our guide to measuring translation quality.
And if you’re evaluating machine translation performance or considering how trained MT engines could support your multilingual content, talk with a language access expert to discuss the right approach for your organization.
