Developers
June 22, 2020

Google Translate Makes Significant Headway Thanks to Machine Learning

Google is using machine learning to continually improve Google Translate, as its latest update demonstrates.
Source: Pixabay

Aside from warp drive and transporters, few technologies featured in Star Trek were more useful than the universal translator, a technology that allowed Starfleet crews to instantly understand and communicate with alien species.

While Star Trek lives firmly in the realm of science fiction, that hasn’t stopped many of the ideas it pioneered from becoming a reality. Live translation software is one of those ideas.

From the beginning, Google has been on the forefront of live translation, with its Google Translate product. A recent update brings welcome improvements, thanks in large part to machine learning.

The Evolution of Google Translate

When Google Translate first appeared, it relied on a phrase-based translation model. Unlike word-based translation, where the base unit is a single world, phrase-based translation models focus on entire word sequences. This helps the model deliver greater accuracy than word-based translation algorithms.

Unfortunately, phrase-based translation has limitations, especially on the engineering end. As a result, in 2016, Google introduced Neural Machine Translation. In the blog post announcing the release, the Google Brain research team said the following:

“Whereas Phrase-Based Machine Translation (PBMT) breaks an input sentence into words and phrases to be translated largely independently, Neural Machine Translation (NMT) considers the entire input sentence as a unit for translation. The advantage of this approach is that it requires fewer engineering design choices than previous Phrase-Based translation systems. When it first came out, NMT showed equivalent accuracy with existing Phrase-Based translation systems on modest-sized public benchmark data sets.”

Despite the promise of NMT and the benefits it brought to translation, it still didn’t bring live translation software up to par with a professional human translator. This was especially apparent when dealing with languages that have small data sets, or low-resource languages. For example, live translation systems do well with common languages like German or Spanish, languages that have an abundance of data to train the NMT system with. More obscure languages, however, pose a problem for the software, as there is not enough training data to help it become proficient.

Current Advances

Google has been working hard to improve on the above limitations, relying on a number of methods to do so.

One such method is Hybrid Model Architecture, combining the best elements of different types of neural network architecture, such as recurrent neural networks (RNN) and the company’s Transformer model, based on a self-attention mechanism.

“Four years ago we introduced the RNN-based GNMT model, which yielded large quality improvements and enabled Translate to cover many more languages,” writes Jakob Uszkoreit, Software Engineer, Natural Language Understanding. “Following our work decoupling different aspects of model performance, we have replaced the original GNMT system, instead training models with a transformer encoder and an RNN decoder, implemented in Lingvo (a TensorFlow framework). Transformer models have been demonstrated to be generally more effective at machine translation than RNN models, but our work suggested that most of these quality gains were from the transformer encoder, and that the transformer decoder was not significantly better than the RNN decoder. Since the RNN decoder is much faster at inference time, we applied a variety of optimizations before coupling it with the transformer encoder. The resulting hybrid models are higher-quality, more stable in training, and exhibit lower latency.”

Similarly, Google has also worked to improve the quality of the training data the neural network relies on. While the Neural Machine Translation (NMT) models learn from data collected from the web, Google has refined their web crawling process to focus on higher-quality data.

Another significant method Google is using is M4 Modeling. This is a process where a single model is used to translate between all languages and English. This helps the translation process for low-resource languages, as it occurs side-by-side with high-resource language translation. As a result, the neural network can then learn from the process at large, improving how it handles the translation of those low-resource languages.

One example of the progress Google Translate has made is how the models handle a string of characters.

“The new models show increased robustness to machine translation hallucination, a phenomenon in which models produce strange ‘translations’ when given nonsense input,” continues Uszkoreit. “This is a common problem for models that have been trained on small amounts of data, and affects many low-resource languages. For example, when given the string of Telugu characters ‘ష ష ష ష ష ష ష ష ష ష ష ష ష ష ష’, the old model produced the nonsensical output ‘Shenzhen Shenzhen Shaw International Airport (SSH)’, seemingly trying to make sense of the sounds, whereas the new model correctly learns to transliterate this as ‘Sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh’.”

The Future

Overall, Google has made impressive improvements and advances in the realm of translation. While it may not yet achieve the standard set by Star Trek’s universal translator, Google’s use of machine learning models is a significant step in the right direction.

TagsMachine LearnigGoogle Translate
Matt Milano
Technical Writer
Matt is a tech journalist and writer with a background in web and software development.

Related Articles

Back
DevelopersJune 22, 2020
Google Translate Makes Significant Headway Thanks to Machine Learning
Google is using machine learning to continually improve Google Translate, as its latest update demonstrates.

Aside from warp drive and transporters, few technologies featured in Star Trek were more useful than the universal translator, a technology that allowed Starfleet crews to instantly understand and communicate with alien species.

While Star Trek lives firmly in the realm of science fiction, that hasn’t stopped many of the ideas it pioneered from becoming a reality. Live translation software is one of those ideas.

From the beginning, Google has been on the forefront of live translation, with its Google Translate product. A recent update brings welcome improvements, thanks in large part to machine learning.

The Evolution of Google Translate

When Google Translate first appeared, it relied on a phrase-based translation model. Unlike word-based translation, where the base unit is a single world, phrase-based translation models focus on entire word sequences. This helps the model deliver greater accuracy than word-based translation algorithms.

Unfortunately, phrase-based translation has limitations, especially on the engineering end. As a result, in 2016, Google introduced Neural Machine Translation. In the blog post announcing the release, the Google Brain research team said the following:

“Whereas Phrase-Based Machine Translation (PBMT) breaks an input sentence into words and phrases to be translated largely independently, Neural Machine Translation (NMT) considers the entire input sentence as a unit for translation. The advantage of this approach is that it requires fewer engineering design choices than previous Phrase-Based translation systems. When it first came out, NMT showed equivalent accuracy with existing Phrase-Based translation systems on modest-sized public benchmark data sets.”

Despite the promise of NMT and the benefits it brought to translation, it still didn’t bring live translation software up to par with a professional human translator. This was especially apparent when dealing with languages that have small data sets, or low-resource languages. For example, live translation systems do well with common languages like German or Spanish, languages that have an abundance of data to train the NMT system with. More obscure languages, however, pose a problem for the software, as there is not enough training data to help it become proficient.

Current Advances

Google has been working hard to improve on the above limitations, relying on a number of methods to do so.

One such method is Hybrid Model Architecture, combining the best elements of different types of neural network architecture, such as recurrent neural networks (RNN) and the company’s Transformer model, based on a self-attention mechanism.

“Four years ago we introduced the RNN-based GNMT model, which yielded large quality improvements and enabled Translate to cover many more languages,” writes Jakob Uszkoreit, Software Engineer, Natural Language Understanding. “Following our work decoupling different aspects of model performance, we have replaced the original GNMT system, instead training models with a transformer encoder and an RNN decoder, implemented in Lingvo (a TensorFlow framework). Transformer models have been demonstrated to be generally more effective at machine translation than RNN models, but our work suggested that most of these quality gains were from the transformer encoder, and that the transformer decoder was not significantly better than the RNN decoder. Since the RNN decoder is much faster at inference time, we applied a variety of optimizations before coupling it with the transformer encoder. The resulting hybrid models are higher-quality, more stable in training, and exhibit lower latency.”

Similarly, Google has also worked to improve the quality of the training data the neural network relies on. While the Neural Machine Translation (NMT) models learn from data collected from the web, Google has refined their web crawling process to focus on higher-quality data.

Another significant method Google is using is M4 Modeling. This is a process where a single model is used to translate between all languages and English. This helps the translation process for low-resource languages, as it occurs side-by-side with high-resource language translation. As a result, the neural network can then learn from the process at large, improving how it handles the translation of those low-resource languages.

One example of the progress Google Translate has made is how the models handle a string of characters.

“The new models show increased robustness to machine translation hallucination, a phenomenon in which models produce strange ‘translations’ when given nonsense input,” continues Uszkoreit. “This is a common problem for models that have been trained on small amounts of data, and affects many low-resource languages. For example, when given the string of Telugu characters ‘ష ష ష ష ష ష ష ష ష ష ష ష ష ష ష’, the old model produced the nonsensical output ‘Shenzhen Shenzhen Shaw International Airport (SSH)’, seemingly trying to make sense of the sounds, whereas the new model correctly learns to transliterate this as ‘Sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh sh’.”

The Future

Overall, Google has made impressive improvements and advances in the realm of translation. While it may not yet achieve the standard set by Star Trek’s universal translator, Google’s use of machine learning models is a significant step in the right direction.

Machine Learnig
Google Translate
About the author
Matt Milano -Technical Writer
Matt is a tech journalist and writer with a background in web and software development.

Related Articles