Picturing the imaginary

Whether or not you think generative AI belongs on screens, you’ll probably find it useful in pre-production.

Hi everyone,

Summer is here, and we are back with a new edition.

This time, our Chief Creative Officer Lars makes the case for generative AI in pre-visualisation and, more generally, for organising the entire production pipeline around generative AI. After that, we’ll reveal how his entry fared at Runway’s “Big Pitch Contest” last month.

Director’s Cut (with our CCO Lars): Picturing the imaginary

You’ll have heard a lot of people opine on what AI can and can’t do for film. These discussions often centre around generative AI that eventually reaches the screen – be it a little post-production fix or fully virtual cinema. But wherever you stand on that, it is important to bear in mind that generative AI will affect other stages of production, too.

Most obviously perhaps, this concerns all the repetitive tasks that precede the shoot. Just think of the progression from scripts to shot lists to shooting schedules and call sheets. This process could certainly be expedited with dedicated tools, even if professional supervision is still needed. (We happen to be working on this.)

Today, however, we are going to look at a different area where generative AI could play a meaningful role: pre-visualisation.

A central problem of pre-production has always been to convey the images in your head to others – through pictures, words, or otherwise. Traditionally, you would use storyboards and explain your desired ‘look’ verbally. But it seems straightforward that AI-generated imagery and footage could come in here.

Don’t take it from us, take it from Martin Scorsese, who has just joined Black Forest Labs as an adviser. According to Scorsese, the biggest problem during pre-production is communication: We lack efficient means of materialising mental imagery and conveying it to others. Of course, you could sketch something and stitch together a few references. But for a realistic picture, and even more so for a moving scene, you cannot actualise the vision in your head before the camera rolls. A tautological point, it seems: You don’t have the film before you make the film.

To some degree, this is about to change. Given sufficient control over camera moves, framing, lighting, lens choice etc., directors or dedicated artists could create storyboard panels and previs shots at scale. And luckily, tools are increasingly allowing for such professional control.

Admittedly, even when used in pre-visualisation, generative AI may face resistance from within the industry. For every Scorsese there is a Guillermo del Toro, a staunch opponent of letting AI anywhere near film. Even in pre-visualisation, he might say, image and video models only recombine historical patterns and impede the fantasy of the creatives involved.

But firstly, mining the history of film for references has been common practice for decades, and today many creatives are already using platforms like ShotDeck for cinematic stills. Secondly, control over synthetic outputs is increasing while barriers to usage are decreasing. And thirdly, we are going to see ever more tools and workflows that are tailored to specific pre-production tasks like pre-visualisation. Today, ChatGPT Images 2 is already creating storyboard-style panels, even though there is certainly room for improvement. And you could go further: Imagine a dedicated LoRA that is fine-tuned on a specific director’s work to match her style. Over time, these visual references may become a new industry standard.

Speaking more generally: Image and video models have matured enough for us to rethink the production pipeline and organise it around the fact that you can create visuals without a camera. This would mean that the role of AI studios won’t be restricted to post-production. Rather, they are likely to become a companion throughout the entire production pipeline, starting with development.

Just consider all the areas where you might already want to have AI specialists on your next project:

• Development: Have an expert on your team who knows what generative AI can and can’t do. Some things that used to be beyond your budget may be affordable now. To be sure, difficult set extensions and creature work still have a price tag – but a lower one.

• Pitch and financing: Mood reels that closely match your vision would be a valuable addition to your next pitch to a broadcaster or fund.

• Previs (moving shot list): Animatic shots or scenes bypass long-winded explanations and get everyone on the same page more swiftly.

• Lensing and lighting: Visualise lens choices and lighting moods during pre-production and agree on the essentials early on.

• VFX reference: Define visual targets based on rough-and-ready footage, not storyboards and text. This helps to work out reliable price quotes and saves everyone time.

Notice that in all these cases, many of the usual worries (resolution, consistency, EU AI Act compliance etc.) do not apply. They could be useful without most of the downsides and limitations.

So we urge producers to stop focussing on questions like: Are video models ready for this post-production task? Instead, they should begin to ask: Where across my entire production pipeline could generative AI add value?

Update: Runway’s “Big Pitch Contest”

Lastly, an update on our CCO Lars’s entry in Runway’s “Big Pitch Contest for Shows that Don’t Exist Yet.” His film “Chances,” which he made alongside our day-to-day projects, placed 8th – pretty remarkable, we think, considering that this is a global competition that countless well-known AI filmmakers participated in.

Here it is again, in case you’ve missed it:

With that, we wish you a pleasant beginning of summer!