Generating video with AI typically involves typing in a prompt – and then starting over if the AI isn’t quite right. But software giant Adobe is building an AI that can adjust videos as they are being generated, allowing the user to change the camera angle or move objects around in the scene.
Adobe MotionStream is an experimental AI video creator that gives users real-time control over elements in the scene. The experimental technology allows users to change the camera angle and direct the movement in the scene using drag-and-drop style tools, along with sliders.
MotionStream, which was unveiled on April 10, generates the video in pieces in order to allow users to interact with the video near real-time.
All of this happens, Adobe says, as the video is being generated. MotionStream begins with a text prompt, but then offers “immediate visual feedback,” Adobe says the user can then move around the camera for a different angle, or move objects in the scene, watching the results adjust as those changes happen.
Adobe says that MotionStream is also designed to replicate more natural physical movements, which has historically been a challenge for AI.
“That’s where a lot of the magic happens—in the secondary effects that are really hard to control manually,” said Adobe Senior Principal Scientist and MotionStream researcher Eli Shechtman. “If you want to move an elephant, for example, you can click and move its body, but it’s a lot of work to manually make those movements look natural.”
“This currently requires skills and specialized software to rig, and animate or keyframe the animation, following a process that typically takes hours, if not days, depending on scope,” Shechtman added. “Instead, the underlying video generator behind MotionStream is basically simulating the world in real time. So, the elephant’s legs move naturally, and the ears flap naturally as the elephant moves. The model provides you with knowledge about the world, and you can interact with it.”
Generating videos with AI typically involves a wait, but MotionStream actually started from Adobe Research efforts to speed up the process by breaking it down into pieces. Users see the first piece while the latter pieces are being generated in the background.
“The natural next step, once we started breaking videos into pieces,” says Adobe Senior Research Scientist and MotionStream collaborator Richard Zhang, “was to ask for feedback from users as the video is being generated. That’s what brought us to MotionStream. It’s the fruit of a long line of research.”
Once the researchers overcame the delay, they worked to give users the tools to make adjustments, allowing users to interact using tools as the video is being generated to fine-tune the results.
MotionStream is, for now, only an experimental program, but Adobe is now sharing a preview of the program to the public.
You may also like
Browse the best video editing apps.
