ElevenLabs, the artificial intelligence voice startup has reached $1 billion unicorn status thanks to a fresh $80 million investment. Its synthetic voices are among the most natural on the market.
The British company offers a range of free and paid-for AI plans, launching its first product as a beta test in January last year, it has added commercial partnerships and new products.
In addition to the ability to clone any voice from a few minutes of sample audio, ElevenLabs includes a voice library with a range of accents, speaking styles and nationalities.
While its main use is in creating content for education and entertainment purposes, the company plans to roll out new products, including dubbing and translation for films, creating transcriptions and even allowing people to commercially license their voice clones.
To get a better understanding of how well ElevenLabs works at creating custom voices I had ChatGPT write a short radio play about the first human mission to Mars. For timing, I created a scene involving an issue on the spacecraft before launch. It all took about 30 minutes, which is quite fast.
How well does ElevenLabs work?
I presented my drama in the style of an old-time radio show, complete with announcer and classic sci-fi sounds. All the voices were selected from the ElevenLabs library and I used text-to-speech rather than voice-to-speech.
The music was created using Google’s MusicFX experiment but the sound effects used were all from Freesound.org under a public domain licence. Editing was done using Podcastle, an impressive AI-heavy audio editing platform that has its own synthetic voice tools.
The audio quality and the natural tone of the voices were impressive. The problem with ElevenLabs is there is no easy way to inject emotion or performance direction. That is one of the reasons they launched the voice-to-voice model, as it replicates real inflection.
Launch of Dubbing Studio and voice-to-voice
One product that uses voice-to-voice is Dubbing Studio, the first product from ElevenLabs specifically focused on the commercial video market. This is because it requires a greater level of realism. Text-to-speech tools like the one I used are great for commercial and advertising.
The Dubbing Studio can clone each voice within a movie or TV show and use that cloned voice to say the same words in the same tone but in a different language.
It also allows for easy synchronization or injection of audio clips to ensure the transcribed audio is perfectly aligned to what is on screen.
Impact on voice actors
For my show, if I wanted to improve the realism I could hire an actor to voice all the lines and then use ElevenLabs voice-to-voice tool to give a different sounds to each character.
This is in part what the SAG-AFTRA actors were concerned about during the strike last year. Rather than a studio hiring a team of actors, they can pay one person to voice an entire radio drama, cartoon or — potentially most likely, a video game.
Tools like Dubbing Studio could also put an end to work for many actors in non-English speaking countries, or those putting voice to anime or other imports.
The flip side, for listeners, is a much more rapid turnaround of dubbed shows and a cheaper way to convert audio books into audio dramas. The challenge is finding a way to ensure human actors have their voice rights protected and compensated for its use.
