From Story to Film: Building Your Practical AI Filmmaking System

AI video generation has changed dramatically.

With models such as Seedance, MiniMax, and Wan becoming increasingly capable, generating a convincing cinematic shot is no longer the biggest challenge.

Characters can look realistic. Environments can be highly detailed. Visual effects can be impressive. And video models are becoming increasingly good at understanding camera and shot descriptions.

But making one impressive shot is very different from making an entire film.

A film might contain dozens of independently generated shots. Each generation is essentially an isolated event: the model doesn’t truly know what happened in the previous clip.

That creates a much bigger problem:

How do we make independently generated clips feel like one continuous film?

• Characters may change.
• Locations may drift.
• Props may disappear or change.
• Characters may suddenly move to different positions.
• The visual atmosphere may become inconsistent.

And even when every individual shot looks good, the result can still feel like a collection of unrelated clips rather than a film.

So instead of asking:

How can I generate better AI videos?

I started asking a different question:

How can I build a repeatable system that turns a story into a coherent film?

This article is the first version of that system.

I approach AI filmmaking much like traditional filmmaking: start with the story, then progressively transform it into something that can actually be produced.

The workflow I currently use is:

More specifically:

The important idea is that video generation comes relatively late.

Before generating a single clip, we first establish the story, the visual assets, and the overall shot structure.

That turns AI video generation from a sequence of isolated generations into part of a larger production system.

I used this workflow to make the film demonstrated at the beginning of this article.

The final film is around 2.5 minutes long.

What surprised me most wasn’t the final duration.

It was the amount of footage I actually had to generate.

I generated only six 30‑second video segments, and almost every generated segment could be used in the final edit.

For an AI filmmaking workflow, that’s important.

One of the biggest hidden costs of AI filmmaking is not simply the price of generation. It’s the number of failed generations you have to throw away.

A workflow that reduces that waste can make a huge difference.

So how did I do it?

Let’s go through the system step by step.

The first step is the story.

But there is an important difference between writing a story for a reader and writing a story that will eventually become a film.

One common mistake is to give an AI model a story full of information that exists only inside a character’s mind.

For example:

He feels lonely.

A human reader can understand this immediately.

But a filmmaker has to ask:

How will the audience see that loneliness?

Instead of simply telling the AI:

He feels lonely.

We can describe something that can actually be filmed:

On a quiet rainy night, he walks alone into the distance.

This is essentially the classic filmmaking principle:

Show, don’t tell.

AI filmmaking is still filmmaking.

If something is important to the story, it is much better to express it through things that can actually appear on screen:

• actions
• expressions
• body language
• environments
• objects
• dialogue
• visual events

For example, instead of writing:

She is introverted and uncomfortable around people.

You might write:

She avoids eye contact, answers with only a few words, and quietly steps away from the conversation.

The goal isn’t to force your story into a professional screenplay format.

The goal is to make the story visually explicit enough for an AI filmmaking pipeline to work with it.

Don’t Have a Story Yet?

That’s fine.

You can start with a simple idea.

For example:

A painter keeps dreaming about a woman he has never met. One day, he meets someone who looks exactly like the woman in his painting.

You can then ask a language model to develop that idea into a filmable story while specifying the narrative requirements you care about.

This is usually an iterative process.

I rarely expect the first generation to be the final story.

Instead:

The important thing is to finish the story before moving too deeply into production.

Thanks for reading! Subscribe for free to receive new posts and support my work.

Once the story is ready, the next step is to build the visual world of the film.

I call this the Asset Library.

Instead of asking an AI video model to invent everything independently for every shot, we first establish the important visual elements:

The asset library becomes the visual foundation of the entire film.

You can think of it as a visual source of truth.

Step 1: Extract the Assets

First, I use a large language model to read the entire story and identify the assets that will need to remain consistent.

The model extracts:

• Characters
• Locations
• Props

For each asset, it also generates a detailed description and an image‑generation prompt.

I prefer having the model output the information in structured JSON, because it makes the assets easier to manage and reuse later.

Step 2: Generate the Reference Images

Different types of assets require different reference strategies.

Characters

For important characters, I generally create:

• a three‑view character reference
• a close‑up / half‑body portrait

The three‑view reference helps establish the character’s overall appearance, while the close‑up provides more detailed facial information.

For image generation, I currently tend to use models such as GPT Image and Nano Banana because they are capable of following structured visual instructions and generating detailed references.

For video generation, character reference methods are a more complicated problem.

Some video models, such as Seedance, apply additional processing or verification to reference faces. There are different techniques for dealing with this.

One method I have been experimenting with is converting part of the face into line art before using it as a reference.

But I don’t consider this problem solved.

Which reference method produces the most consistent character identity?

That’s a research question I want to test systematically in a future article.

Locations

For locations, I care less about exact character‑like consistency and more about establishing a coherent visual language.

This is where stylized image generation can be particularly useful.

I currently find Midjourney useful for establishing the overall visual style of environments.

The environment is not just a background.

It helps establish the tone and atmosphere of the entire film.

Props

Props are generally simpler.

For important objects, I create dedicated reference images so that the same object can be referenced across different shots.

This is especially useful when an object is important to the story.

This is the most important part of the workflow.

The goal here is to transform the story into a sequence of shots that can actually be generated as video.

I call this component the:

Director Agent

Its job is not simply to write video prompts.

It first needs to understand the entire film.

The process has three major stages:

The Mistake I Made Initially

When I first built the workflow, I tried something that seems intuitive:

The problem was that the resulting clips often felt like independent parallel worlds.

Each segment was coherent on its own.

But the relationship between segments was weak.

Why?

Because the AI had no real understanding of the entire film.

It was solving each small section independently.

Plan the Whole Film First

I changed the workflow.

Instead of segmenting the story first, I ask the AI to plan the entire film storyboard first.

This small change made a surprisingly large difference.

The model first understands the entire narrative and creates a long sequence of shots.

Only after that do I divide those shots into individual video‑generation segments.

This gives the shots a much stronger sense of continuity.

The AI is no longer designing six independent mini‑films.

It is designing one film that happens to be generated in multiple pieces.

The Three‑Step Director Workflow

Step 1 — Plan the Entire Storyboard

First, the AI reads the complete story and creates a complete storyboard list.

At this stage, the descriptions don’t need to be extremely detailed.

The goal is to establish:

• what happens
• which characters are present
• where they are
• what the camera sees
• how the story progresses visually

Step 2 — Segment the Storyboard

Once the complete storyboard list exists, I divide it into video‑generation segments.

The boundaries can come from:

• the model’s maximum generation duration
• natural story transitions
• changes of location
• changes of scene
• moments where a new visual sequence begins

But importantly:

I am segmenting the storyboard, not independently storyboarding each segment.

This preserves the larger cinematic structure.

Step 3 — Refine the Connections

The initial storyboard is still only a plan.

After segmentation, I ask the AI to refine the individual shots and, especially, the transitions between segments.

The goal is to make the end of one segment naturally lead into the beginning of the next.

Sometimes this can be solved through shot design alone.

Sometimes I use the last frame of the previous video as the first‑frame reference for the next video.

But this is not necessary for every transition.

If the camera language already connects naturally, or if the next shot is an establishing shot or another intentional transition, forcing a first/last‑frame connection may not provide much benefit.

The important principle is:

Use continuity tools when they solve a continuity problem — not simply because they are available.

The Video Prompt Structure

After these steps, each video segment receives a structured prompt.

My current structure looks roughly like this:

The exact format can be adapted to different video models.

The important thing is the separation of responsibilities.

The total description establishes the overall scene.

The shot breakdown controls the individual shots.

The reference declaration tells the model which visual assets should remain consistent.

Why This Works

At this point, the video model is no longer receiving something like:

A man walks into a cafe and talks to a woman.

Instead, it receives a much richer production package:

This doesn’t eliminate AI generation uncertainty.

But it significantly reduces the number of things the model has to invent from scratch.

And that is one of the central ideas behind this workflow:

The more important information you can establish before generation, the less you have to leave to chance during generation.

Once the asset library and video prompts are ready, the actual generation process becomes relatively straightforward.

Select your preferred generation platform and model, copy the final prompts for each scene generated by the Director Agent, and attach the corresponding asset reference images in sequential order to generate each clip one by one. Because these clips are built on an organized production system, you will ideally receive a series of continuous shots that follow the narrative progression seamlessly. However, the output is rarely perfect, and manual adjustments or refinements will still be required.

For high‑quality cinematic results, I currently prefer Seedance 2.5 for this workflow, although the exact model will inevitably change as the technology develops. The important thing is that the workflow is model‑agnostic; the system should not depend on one particular video model. The model is the generator—the workflow is the production system.

Generating the clips is not the end.

It is only when the clips are edited together that they become a film.

Editing is a much deeper subject than I can cover in this article, and I’m still exploring it myself.

But there are a few principles that have already proven useful.

Sound Can Connect Shots

Sound design is incredibly useful for transitions.

Sometimes the next scene’s sound can begin before the image changes.

The audience hears the next environment first, and then the visual follows.

This creates a smoother transition between otherwise separate AI‑generated clips.

Don’t Be Afraid to Reorder Shots

The order generated by the AI isn’t always the best final order.

Sometimes simply moving shots around can improve the rhythm or narrative clarity.

AI gives you raw material.

The editor still makes the film.

Use Montage

Montage can be especially powerful in AI filmmaking.

A sequence of visually different shots can feel surprisingly coherent when they share:

• music
• sound
• thematic meaning
• visual atmosphere

A continuous piece of music can sometimes connect shots that would otherwise feel disconnected.

This is one of the reasons editing remains so important even when the generation quality is extremely high.

Using this workflow, I generated six 30‑second video segments and edited them into a film of approximately 2.5 minutes.

Most importantly, I was able to use essentially every generated segment.

That is not a claim that the workflow produces perfect films.

It doesn’t.

But it demonstrates something I care about much more:

A structured AI filmmaking workflow can significantly reduce the amount of randomness involved in turning a story into a film.

I don’t want to present this workflow as a finished solution.

It isn’t.

There are still several major problems.

1. Context Length

The overall storyboard and segmentation process requires a language model with strong reasoning capabilities, high context length, and reasonable generation speed.

In my current experiments, a single pass can handle roughly a 10‑minute story, around 1,600–1,800 words, while consuming more than 10K tokens.

Longer films will require a better hierarchical planning strategy.

2. Spatial Consistency

This is probably one of the biggest remaining problems.

Characters and objects can maintain their visual identities while still appearing in completely different spatial relationships between shots.

For example:

Character A is standing to the left of Character B in one shot.

Then in the next shot:

Character A somehow appears on the other side.

The individual images may look fine.

The scene doesn’t.

My current workflow doesn’t have a strong solution for this yet.

Everything described here can be built manually.

You can use your preferred language model, image model, video model, and editing software and reproduce the workflow yourself.

That’s important to me.

The goal of this publication isn’t simply to tell you:

Use my product.

It’s to figure out:

What actually works?

And then make those methods reproducible.

I’ve included the prompts used throughout this workflow so you can experiment with them yourself.

Download Prompt Package

This prompt package is designed for maximum simplicity: simply attach the TXT file, enter your prompt or the required input, and let the agent handle the rest. Be sure to review the included instructions before getting started.

You can also adapt the workflow to whatever AI tools you already use.

I’ve been building TikFilmer alongside this research.

The idea behind TikFilmer is simple:

Take the filmmaking workflow described above and turn it into a practical tool.

The research comes first.

When I encounter a problem while making a film, I try to understand it.

Then I experiment.

When I find a method that works, I try to turn it into part of the system.

And eventually, that becomes a feature in TikFilmer.

So the relationship is:

That is also the philosophy behind this publication.

I don’t think we have figured out AI filmmaking yet.

In fact, I think we’re still very early.

The models are getting better incredibly quickly.

But better models alone don’t automatically solve the filmmaking problem.

We still need better ways to plan stories, build visual worlds, control space, preserve characters, design shots, manage continuity, and edit everything into something that actually feels like a film.

That’s what I want to explore here.

Not just:

What can AI generate?

But:

How can we build a repeatable system for making films with it?

If you’re interested in figuring that out too, subscribe.

I’ll document the experiments, failures, workflows, prompts, and methods as the system evolves.

Make films. Solve problems. Build the system. Make better films. Repeat.