How AI Video Generation Works: From Noise to World

How does AI video generation work? It converts a text or image prompt into frames through an iterative denoising process. The model starts with pure noise, then removes that noise step by step until coherent video emerges. It does not animate frame by frame in sequence. Instead, it denoises thousands of pixels simultaneously, guided by your prompt, through 20 to 50 steps per generation. That distinction matters if you are building interactive worlds on top of it.

I ran twelve environment variations through three different engines last month for a walking-sim project. Here is what I learned about how the engine actually runs.

What the prompt actually triggers

Your text prompt does not get handed to a camera. It gets converted into a numerical representation called a conditioning vector, which steers the denoising process from start to finish.

A language encoder first parses your prompt into semantic embeddings. Those embeddings condition a diffusion model, which takes a field of random Gaussian noise and progressively removes it across 20 to 50 denoising steps. Each step produces a slightly cleaner prediction of what the final video should look like.

The video is not generated at full resolution throughout this process. Modern engines work in compressed latent space first, a lower-dimensional representation of the video that dramatically reduces compute cost. A variational autoencoder expands the latent video back to pixel space at the very end.

This architecture is why a 30-second clip costs roughly 10 times more than a 3-second clip on the same platform. Duration scales the latent space, and the model's work scales with it.

A second implication: your prompt words are not instructions. They are probabilities. The model learned statistical associations between language and visual patterns from millions of training videos. When you write a foggy 1920s Detroit jazz club where the bartender is a robot, it activates a cluster of learned correlations, not a blueprint. Two identical prompts will produce two different outputs. That is not a bug. It is the denoising stochasticity working as designed.

Why diffusion won over everything else

Three approaches competed to become the standard in AI video generation: autoregressive models, GANs, and diffusion models.

Autoregressive models predicted video one token at a time, like a language model generates text. Coherent in principle, but slow, and spatial consistency was difficult to maintain at scale. GANs pitted a generator against a discriminator in an adversarial training loop. Fast inference, but notoriously unstable to train and prone to mode collapse. Diffusion models started from noise and learned to remove it iteratively. Slower inference than GANs initially, but they scaled predictably with compute. That scalability won the argument.

The architectural upgrade that defined the current generation happened in 2023 when researchers replaced the U-Net backbone in latent diffusion with a transformer operating on patches. That paper is the architectural ancestor of every serious video model in production today, including Veo 3, Sora 2, and Kling 2.6.

A Diffusion Transformer, or DiT, cuts video into spatiotemporal patches, flattens them into a sequence, and runs transformer attention over that sequence globally. Because attention operates across all patches at once, the model maintains spatial coherence far better than convolutional approaches. The tradeoff is quadratic cost: processing twice as many frames requires four times the attention computation. This is the structural constraint that keeps a single DiT from generating ten-minute videos without additional architectural changes.

The practical output of this architecture: 8 to 20 seconds of video with plausible physics, consistent lighting, and coherent motion, when conditions are right. Two years ago, AI video meant five-second clips with melting hands and drifting geometry. The architecture shift is why that changed.

Text in, image in, video in: three different bets

Most engines now accept three input modes. They produce meaningfully different results for anyone building world environments.

Text-to-video offers maximum creative range and minimum control. You describe a scene, the model generates everything, and small prompt variations can produce wildly different outputs. Good for exploring visual styles or generating concept environments quickly. Difficult for maintaining consistency across multiple shots.

Image-to-video starts from a still frame as a visual anchor. The model animates forward from that reference, preserving the exact look of the source. This is the most controllable path available today. The workflow is practical: generate a key frame with a high-quality image model, then animate it. You get consistent visual language, controlled composition, and motion layered on top of a stable base.

Video-to-video transforms an existing clip, changing style, motion intensity, or environmental details while preserving the original structure. Useful for post-processing generated environments rather than generating them from scratch.

For anyone building interactive environments: image-to-video is your fastest path to a repeatable result. Text-to-video is for the exploration phase, before you commit to a visual language.

The coherence cliff no one warns you about

Every DiT-based model has a coherence ceiling. It appears somewhere between 8 and 20 seconds of generated video. Before it, physics hold, the subject stays consistent, the camera movement reads as intentional. After it, temporal artifacts appear: a hand that changes shape mid-motion, a background that shifts between frames, a face that drifts to a different structure.

This is not a quality failure. It is a structural limitation of quadratic attention. Maintaining global temporal context becomes exponentially expensive as duration increases. Current production models can sustain coherence for clips in the 10-to-20-second range at high quality. Beyond that ceiling, even the best engines need architectural workarounds like sliding window attention, which breaks the full temporal context into overlapping chunks at the cost of global coherence.

For world-building pipelines, the implication is direct: generate short clips, stitch them deliberately, and use a consistent reference image to maintain visual continuity across shots. A single prompt will not give you a coherent two-minute environment sequence. The models that claim otherwise are averaging over many failed attempts you do not see.

The workaround that holds together in production: treat each generated clip as one tile in a larger mosaic. Define a key-frame image as the visual anchor, generate 5-to-8-second clips from that anchor, and assemble them. Slower than a single generation. Also the only approach that produces a coherent game scene rather than a drift artifact.

Where world generation diverges from flat video

Standard AI video generation produces passive video: a sequence of frames you can only watch. Interactive world generation asks something the flat-video pipeline was not designed for: can the space respond to what the player does?

The gap is architectural. A flat-video model generates a fixed tensor of frames. A world model must generate a latent space that remains navigable, forkable, and responsive to player input.

This requires training on different data entirely. Flat-video models train on film, television, and captured footage. World models train on gameplay footage, physics simulations, and first-person navigation data. Research projects like Google's Genie and Meta's WorldGen both take this approach, treating the environment as a world model in the reinforcement-learning sense rather than a video-prediction problem.

The practical difference is concrete. In a flat-video tool, you cannot walk left when the generated camera moves right. In a world-generation engine, left and right are both valid continuations of the same model state. The engine generates the consequence of your action, not just the next predicted frame.

Skip flat-video tools if what you need is a traversable space. They produce footage that looks like a world. It will not behave like one.

Open weights, closed APIs, and what that gap actually costs

Two ecosystems have formed around AI video generation, and they serve different types of creators.

Closed commercial APIs including Sora 2, Veo 3, Kling, and Runway offer better quality at the high end and straightforward access at per-second pricing. Open-weight releases including WAN 2.0, HunyuanVideo, and CogVideoX offer full control and no recurring cost, but require real GPU infrastructure to run.

The quality gap between the two tiers has narrowed considerably since early 2025. For a solo creator without a data center, the closed APIs make practical sense for final output. Open weights make sense for fine-tuning on a specific world aesthetic, or for pipelines running at scale where per-second pricing compounds quickly into a real budget problem.

One number worth tracking before committing to a pipeline: generation cost per usable second. Closed APIs currently average $0.05 to $0.20 per generated second depending on resolution and model. At 5 seconds per environment tile and 40 tiles for a short game, that is $10 to $40 in generation cost before any curation or retakes. Include a 30 percent failure rate on generations that do not meet quality standards, and the real number climbs.

Budget before you generate. The cost structure of video generation is not like image generation, where failed attempts are cheap. Each bad take costs real credit.

Which engine fits what you are building tonight

If you have never run a video generation pipeline: start with image-to-video. Generate a single key frame for your world with a strong image model, then animate it. This gives you immediate feedback on motion quality without spending credits on failed text-to-video attempts.

If you are building an environment pipeline for a playable game: pair a high-quality image model for frame generation with a video model for motion. Text-to-video is for exploration. Image-to-video is for production. The two phases are distinct, and conflating them wastes both time and budget.

If you want a space that responds to player input rather than footage that depicts one: the flat-video tools will not get you there. The gap between video that looks like a world and a world you can move through is not a prompting problem. It is a model architecture problem. The engines built specifically for interactive output train on navigation data rather than footage and generate responsive state rather than fixed frames.

Your prompt is already a place. Whether it stays still or keeps moving is a question of which engine you put it through.