What Midjourney did for still images and ChatGPT did for writing, a new crop of apps is aiming to do for video — spitting out animated clips based just on a prompt.
Unlike full AI video editing apps such as Runway Gen-1 or script-to-movie programs like InVideo, these new programs don't work off existing videos, but rather create completely synthetic movies.
This is a very nascent field, though — far behind still-image and text generators.
Clips are limited to 4 seconds or less, depending on the service; and they often have a cartoonish or hand-sketched appearance. You won't be fooling your friends with deepfakes from these services any time soon. That said, these synthetic video creators represent an exciting frontier in AI creativity tools — one that developers and artists will be pushing forward quickly in the coming months and years.
What makes the best AI video generators?
Realism and detail are the key factors in synthetic video. None of these apps fully mimic real-life, but they sometimes get close. Runway Gen-2, for instance, can even model shifting lighting and fine movements. Beyond tackling shapes, colors, and movement, apps have to provide enough resolution to make their creations clear.
Most don't do that (resolutions range from 480p to 1080p). And the square format most use seems more set for meme-like online posting than an attempt at movie creation. Of course, at a maximum of 4 seconds, none of these clips approach movie proportions, but some are even shorter — at two seconds.
Sometimes, however, synthetic videos have too much detail with surrealistic hallucinations found neither in real life nor in the prompts that users feed them.
We've seen things like cats whose bodies disappear and eating utensils that melt into people's faces. All that could be cool if it were specified in the prompt, but it wasn't. It appears that there may be a direct relationship between the sophistication of the app and its tendency to hallucinate. Programs that try to provide more detail may also provide surreal details.
Synthetic videos have too much detail with surrealistic hallucinations found neither in real life nor in the prompts that users feed them.
Whatever the app's qualities, the ability to refine results is key. For instance, how well can it understand complex prompts about colors, camera angles, movements, and more? Some apps also provide filters for different stylistic looks.
But however good the app, it may take several ties to get things right: Generative AI yields different results every time. To account for this, we gave each app three tries for each prompt we tested and then chose the best. We also set the apps to their highest quality settings (for attributes such as resolution and framerate). These online tools were all tested in the Chrome browser.
Price matters, too. While one of these apps is free, the others charge either by the length of clips produced or by the number of video-creation prompts. Pricing gets confusing quickly, with billing by week or month, discounts for year-long subscriptions, and the ability to buy supplements to subscriptions.
Unless otherwise noted, we priced services based on the smallest possible commitment--by the month or by the week.
Runway Gen-2 is the company's first product to create entirely synthetic AI video. (Gen-1 uses AI to modify existing videos.) As such, it shows a lot of promise, but also a lot of quirks. The app generates detailed, rich-looking clips in 16:9 widescreen fashion at 1080p resolution, with sophisticated effects. A movie we made of a cat, for instance, shows flickering of its whiskers, trees swaying in the wind, and light and shadow dancing across its face. The images aren't quite realistic, but rather sophisticated renderings — somewhere between live footage and the best videogame cut scenes.
The app can also understand complex prompts: not just "orange tabby cat," but also "orange tabby cat on a white carpet rolling a pink ball of yarn, warm lighting, realistic." Other rival apps such as Gencraft can also understand these specifications, but they show less detail and sophistication in their renderings.
Perhaps as a side effect of how ambitious Runway Gen-2 is, it can also go too far in its creativity. The app had a tendency to wild hallucinations that we didn't see in rivals. In one attempt at rendering the tabby, for instance, the cat was absent, but the ball of yarn sprouted in human hand. In another test, a horse gained a bicycle wheel. Generative AI is a trial-and-error game, though, and we generally got pretty good results within three tries. Although sometimes even the best attempts had mild hallucinations--like an otherworldly glowing white orb that floated between the cat and its ball of yarn.
Each of these goofs cost you — although not too much. Gen-2's pricing is based on credits: Five of them buys a single second of video. New users get 125 free credits, before having to upgrade to monthly plans, starting at $15 for 625 credits that expire at the end of the month. You can purchase additional, non-expiring credits starting at $10 for 1000.
Gencraft is one of the pioneers in the emerging field of pure text-to-video generators. But as such, it has limits. The app was adept at rendering an orange Tabby cat, for instance, and it could understand sophisticated modifiers in the prompt, such as adding a white carpet and pink ball of yarn (and it's relatively fast--rendering its two-second videos in under 30 seconds.) But getting cats right is pretty much a prerequisite for any online creation tool. Gencraft struggled with newer internet memes, such as "Astronaut riding a horse" or "Will Smith eating spaghetti." These images were blocky or distorted: Smith's head seemed to change shape, and his eyes bulged, for instance.
Gencraft does provide some tools to fine-tune your results. You can choose from a collection of 31 styles. Our Tabby appeared radically different in the Cartoon, Oil Painting, Abstract, and Realistic options. ("Realistic" is a bit superfluous, since images tend towards that if you simply opt to not pick a style at all.) You can also specify keywords of topics or attributes you'd like to exclude from a video, such as "cross-eyed" or "blurry background."
Gencraft does have some additional limitations. It's capped at two-second videos that appear in a square format. Rival Runway Gen-2 and the free ModelScope support up to four seconds; and Runway also provides a 16:9 wide format and 1080p resolution (vs. 720p for Gencraft).
Gencraft offers a free tier that allows you 10 video-creation prompts per day, but the clips have watermarks. You can upgrade to 25 watermark-free prompts per day for $3.99 a week, or unlimited prompts for $9.99 a week.
ModelScope Text to Video Synthesis became an internet sensation when it was used to generate the "will smith eating spaghetti" video meme. And you have plenty opportunity to try out your own crazy ideas. Hosted on AI developer hub Hugging Face, ModelScope is free of charge and can produce videos of up to 4 seconds, which is long for this emerging tech.
Of course "free" has a downside: You're at the whim of the site's servers, which often get overwhelmed and can't complete your job on the first (or second, or third) try. Pushing the tool to its highest quality settings (such as framerate) further hurts your chances. Sometimes, you just have to wait for a quieter time to get things done. And Hugging Face doesn't provide storage: You'll have to download your videos right after making them. (Don't wait, or a server glitch might wipe them out.)
ModelScope's clips don't have the high-end detailing of Runway Gen-2's creations (although they also don't have Runway's hallucinatory images). But they have a high-quality cartoony quality rivaling paid providers Gencraft and Vercel. Its rendering of an orange Tabby cat was attractive. Although like all ModelScope subjects, the cat was quite fidgety, jumping all around. Will Smith was also quite frenetic, stuffing pasta into his mouth with bare hands. But, cartoony as it was, the subject was quite recognizable as the famous actor. Gencraft's also-fidgety version was more distorted. Runway Gen-2 created a dreamy, surreal version that looked cool but didn't look much like Smith.
With support from the developer community and a price tag of zero, ModelScope is worth keeping up with as this tech rapidly advances.
Want to know more about using AI for creative work? Here's our breakdown of the 5 best text-to-video AI image generators.
