Tara With Spear

And other new way's we're making AI work for Escape From Planet Omega-12

The biggest challenge with using AI for film making, whether it be one-hindered-percent AI, or a hybrid of techniques fused together using VFX like we’re doing, is getting characters and settings to “perform” in a way that has a naturalistic flow that pulls the audience into the story from shot to shot. No matter what new tools become available to us the heart of cinematic storytelling still beats within the composition, performance, and the cuts. For this reason, I believe that the future of cinema belongs to the editors and visual-effects artists who embrace AI.

My goal for our characters has always been performance control. It doesn’t matter how “real” they look. We can worry about that later. As a pioneer in this new medium my first priorities lie in me being able to get the audience to feel for them as they emote, interact with environments, handle props, and generally melt into the story that’s being told.

Today, I’m going to show you some examples of various AI tools, and the thought processes behind using them to get the results I believe will change how we make movies.

Here we have a few seconds of Tara, one of our protagonists in Escape From Planet Omega-12, awkwardly holding a spear as she steps cautiously through a yet to be composited alien environment; hence the green screen background. She was originally created from a combination of my art, Midjourney character training, and ComfyUI, combined in Photoshop at the end.

As short as it is, I’m very happy with this result. It’s all there; the physics of her hair, clothing, and flesh, the expression of near-comical uncertainty, the poor handling of the spear she’s holding, which was a happy accident, yet totally appropriate to the character. I actually know what chased that and now can recreate it for other shots. Yes, it’s all very promising so let’s talk about each step that got us to this point.

Tara’s spear was an idea I had very early on, something that would transform Tara from just being a beautiful woman to a more mythic silhouette. Like all of my ideas, it started with sketches in my drawing pad.

I then made a quick illustration of the spear in Photoshop.

Once that was done, I imported it into Google Gemini, describing to it in a prompt what it was, what it was made of, and then telling it to give me a realistic version of it, in the form of a style sheet.

From there I asked Gemini for different perspectives on the weapon, which I then upscaled and imported back into Photoshop so I could break each angle into a separate image that I could use to train a “prop” model using KlingO1’s object function.

I repeated his process for her ray-gun, which I had built as a practical effects prop, and her sword which I did as a another Gemini transformed illustration.

From there I uploaded this image of Tara into Kling and prompted it to add the gun and sword to her belt on the side that was angled lower, as if pulled down by their wait, and put the spear in her hand.

Kling’s outputs our video, rather than image, so this is the first result I got. The combination of props proved to be too much for it, though, and it started to melt and morph everything together, as well as exhibit a clear lack of understanding of the object sizes.

I tried to make is simpler by just prompting for one prop at a time, as I also had 3D scans of the other weapons that I could track and composite onto her later.

It still struggled with the spear, though. I used prompts and a few re-rolls to get it right, and got this result. As mentioned before, she’s holding the spear a bit too high up but I didn’t think it was wholly inappropriate for the character, as she has no training or survival skills.

Now, it was time to see if we could get a real performance out of her, rather than just a runway strut, using footage that I shot with actress Isla Cervelli. Taking a still from the above video, I applied it to our reference footage using WAN2.2, an open-source AI model that I’m running locally on my computer.

With no MoCap suit or tracking dots, it took everything Isla did and applied it to Tara, understanding the physics of her body, hair, and even understood the metallic spear reflecting light as it moved, as well as incorporating how she was shown holding it too high. I thought this was a wonderful result when I saw it because, again, Tara wouldn’t know how to hold a spear properly, at least not at the beginning of our story.

But, I still wanted to see if I could get her to hold it properly by changing the reference, so I brought the still back into Photoshop and made it so she was gripping the spear in the middle.

I tried a different shot of Isla, so that might have also influenced it but…

…It worked.

Although, I’m not sure why it decided that hair needed to be in her face. I’m sure that with a little practice and understanding of what to prompt, I can fix that issue. I’m going to be trying other AI models, too. One called LTX-2 just came out which is supposed to be faster and more accurate, so I’ll be downloading and testing that one next weekend.

I’ve also been trying out more proprietary models, like Grok, for incidental moments that might not require as much control.

Some of the results might be usable with some up-scaling and face fixing, here and there, but for true character performance moments we’ll have to do things the “hard way” so to speak.

Lot’s more to talk about, regarding this process but I’ll have to leave it for another time, as my pet’s are all meowing and wining at my office door for dinner. I’ll put a few more examples for you paid supporters after the jump. Please, feel free to leave questions or comments below, regarding the process. It’s still early days but I think the potential in the hands of creative individual is largely self-evident.

Best,

Christopher