The technology is currently in open beta testing.
Russian IT giant Yandex has launched a new service that adds simultaneous translation to YouTube live streams. So now the announcements of the long-awaited new products in the IT industry, interviews with celebrities from the red carpet and inspiring broadcasts of space launches can be watched in Russian immediately, without waiting for the translated versions to appear.
Voice-over translation of streaming video is an incredibly difficult engineering task, so five neural networks were used at once to solve it. The first one recognizes the audio track and turns it into text. The second neural network determines the gender of the speakers, the third cuts the text into sentences – places punctuation marks and selects from the text parts containing a complete thought. Only the fourth neural network directly translates the received pieces, and the fifth already synthesizes speech.
While the technology is in open beta testing, to translate broadcasts on YouTube channels from playlist.
