Adobe has been working on their generative AI initiatives under the Firefly brand for a couple years now, with ongoing improvements and integrations. It started with cloud based image generation in 2023, and grew to include vector and design generation, and now video generation. It also added direct integration in Photoshop for image generation, which works pretty impressively at this point, and they are beginning integration to Premiere Pro with Generative Extend in the current beta application. But their broader AI generated video tools have only been in limited beta form until now.
One of the challenges of generating video, is how does the user communicate to the AI what it is they are looking for, beyond just text? (If a picture is worth a thousand words, I don’t have time to write a proper video prompt;) Adobe has a number of other approaches that users can try, including starting with a still image to work towards or from, and selection choices for camera angle and motion. I imagine that the next step will be to add video as source, and allow AI to modify it, similar to how photoshop allows you to remove objects from selected portions of an image.
I did some testing with my access to the existing generative AI video beta, but there is a learning curve. My initial results were very ‘AI’ looking, people morphing into each other, wakes without boats making them, etc. Starting with an image got me a little closer to what I was hoping for, but still led to some unexpected places, and human figures got distorted. I tried using text to image, and then using image to video, to give me more control over the intermediate processing, which got me on a better track. My best result was from generating a castle image. I originally wanted to see through the gate as the drawbridge was lowering and the portcullis was raising. This was the best image I got, but no drawbridge or portcullis:
I then sent that to the image-to-video tool, and had it close the doors while zooming out. I downloaded that and reversed it in Premiere to get the opening effect. Then, inspired by some AI demo assets of smoke and flame plates, I prompted it to start with the same image, and explode the castle while zooming in. While showing off my amazing AI creations to my young son, because he is into castles, he asked if I could have the castle explode as the doors were opening? Yes I can, because they have the same starting frame, that was a seamless edit, once one was reversed. Here is the result once I sped it up and exported for web:
That is pretty decent for a couple minutes work, starting from nothing. And using the start frame to end frame technique, I could assemble a continuous shot broken into numerous 5 second segments of action, based on a set of storyboards or other generated images. So all sorts of interesting possibilities open up within that framework. The finished result of that experiment was something good enough for pre-viz or post-viz purposes. So I am looking forward to trying it out on a larger project. Maybe starting with AI in Photoshop to modify a frame export, and then use Firefly to make that back into a video, or to make it blow up.
Adobe also announced a tool for translating video of an on screen character’s video from one language to another, while matching voice tone, AND generating lip-synced mouth movements. They have mentioned this before, but I was looking forward to seeing it in action, not that I know any other languages well enough to verify it. But this could be a valuable tool for localization, especially for corporate video type jobs, or for video podcasts and such.
As part of these new features Adobe is launching standalone plans for Firefly: $10/month for the basic Firefly Standard plan, which offers 2000 credits, which is twice as much as are included by default with existing Creative Cloud subscriptions, and $30/month for the Firefly Pro option, which offers 7000 credits for AI processing. It looks like a 5 second video is expected to cost 100 credits. A 5 sec 24p video is 120 frames, so I guess that seems reasonable. There will be a Firefly Premium option available for users with huge AI needs, but the details of how that will work were not announced.
This is just one more step in Adobe’s broader AI efforts, and it will be interesting to see where it leads next. Now this isn’t a simple ‘automatic’ solution to all of your problems, and just like any other new medium or toolset, there is a learning curve, and it takes time to explore and orient yourself to the possibilities, but investigating those options early can help you stay ahead of the curve. So now is a perfect opportunity for existing Adobe users to check out what Firefly now has to offer in the new generative video tools.
