The Yandex team announced the launch of a very convenient feature for users, the ability to watch English-language videos on many popular platforms, including YouTube and Vimeo, with automatic Russian-language voice acting.
In the summer, Yandex began testing machine translation technology for video and offered users a special playlist for demonstration. Last week the new feature is open to mass use. As the developers promise, now users will be able to watch almost any English-language video with Russian voice acting.
Russian voice-over translation is offered in two versions with a male and a female voice. Translation works in Yandex Browser for computers (Windows, macOS, Linux) and Android, as well as in the Yandex application for Android and iOS. To watch a video with Russian voice acting, you need to press a button in the player and wait for a little. The translation process takes a few minutes. When everything is ready, you will receive a notification.
Translation takes place in several stages. First, Yandex recognizes speech, turns it into text, and uses neural networks to break it down into sentences. Then he determines the gender of the speaker, translates sentences into Russian, and synthesizes the voice. Then the translation is combined with the video sequence using neural networks. As the developers note, this is not a trivial task at all, because sentences in Russian are longer than in English.