Making Video With AI: The Model, the Harness and the Loop Are One Stack

Last week NO1ennn’s long post on turning Opus 5.5 into a studio made the rounds on X. Its core idea is strong: code-drawn animation (motion design), the scaffolding around the model (harness) and the cycle that keeps an agent working (loop) get discussed as three trends, but they are three layers of one stack. Here I put that idea in my own words, test it against what happened while producing our own video series, and spell out where I disagree.

I covered the basics of the harness idea in What Is a Harness in AI Agents?; this builds on it.

The model doesn’t hand you a video, it writes a program

A language model takes text and images in and puts text out. It can’t give you an MP4. What it gives you is a program that knows how to draw one frame. The video comes from running that program for every frame and joining the results.

The rule that carries everything fits in one sentence: a frame depends only on its own number. No clocks, no timers, no random numbers that change between runs. Frame 431 looks the same every time. That means:

• Rendering the same video again gives the same result.
• Fixing a mistake is a one-line edit and a re-render.
• You can open and check a single frame without playing the whole thing.

The vertical videos on our own channel are made this way. When each element on the board appears is computed from the real time of the word in the voice-over, so changing a label doesn’t mean rebuilding the video.

The four parts of a studio

The author splits the studio into four parts, and the split really helps:

1. The stage: one canvas at a fixed size, with the fonts embedded so nothing loads from the internet.
2. The renderer: an engine that calls the draw function for every frame, headless, with no screen recording.
3. The encoder: the step that turns frames into a video file (ffmpeg, for example).
4. The checks: the part everyone skips, and the part that makes it look professional.

The model writes the first part; you own the other three. Once they exist, every new video is just a new draw function, not a new studio.

The rules file and the brief

The model gets a rules file it reads on every run. The author’s five rules, in short: a frame depends only on its number; the loop closes (the last frame flows into the first without a jump); colour only where it carries meaning; every label traces to a source, no invented numbers; nothing ships until the checks are clean. Every rule came from a mistake.

The brief decides the result more than the model does:

• Describe the mechanism, not the vibe. “A table of contents that reads itself, one section at a time” beats “make it dynamic.”
• Name every state of an element: idle, live, done, and what each looks like.
• Give it the real material: numbers, lists, source text. Models are good at layout and bad at guessing facts.
• Ask for the plan before the code: which region goes where, what moves when.

A vague brief gets you the default every model falls back to: a centred title on a gradient with everything fading in.

The checks: nobody looks until the counters read zero

In the author’s setup, three checks run after every build:

• A smoke test: the page runs headless for a full loop, and counters for runtime errors, invalid colours, text off the stage, text out of its panel and overlapping text must all read zero.
• A seam check: frame 0 and the frame exactly one loop later must be pixel-identical.
• A look: stills are enlarged and actually read.

We do the same: no text on the board smaller than 32 pixels, zero overflow, subtitles within a fixed width. But there is something important I’d add, below.

The harness: guides and sensors

Birgitta Böckeler’s harness engineering article at Thoughtworks splits control into two kinds:

• Guides steer the agent before it acts: rules files, skill files, reference docs.
• Sensors look after it acts: tests, linters, a second review agent. Some are computational (fast, same answer every time), some inferential (another model’s judgment; more expensive and less certain).

Map the studio onto it and the picture matches: the rules file and brief are guides, the smoke test and seam check are computational sensors, looking at the frames is an inferential sensor. Her warning matters too: no sensor reliably catches a misdiagnosed problem, needless complexity or a misunderstood instruction. If you never said clearly what you wanted, correctness is out of any sensor’s reach.

The September 2026 paper that read the source of eleven coding agents adds an interesting picture: none of them finds code with vector search, all use text search and file patterns, and skill files (9 of 11 systems) have overtaken the external connection protocol MCP (8 of 11).

The loop: keeping the agent working on its own

A harness makes one run better; a loop keeps runs happening. The Claude Code team’s split is useful:

• Every prompt is a loop: the agent gathers context, acts, checks; you judge the result.
• The goal loop: you define what done means; each time the agent wants to stop, an evaluator checks the condition and sends it back.
• The time loop: re-runs a job at an interval.
• Routines triggered by an event or a schedule: they run with no human in real time.

A good goal has three parts: a stop condition a machine can check, a turn limit, and a “no progress for two rounds, stop” rule. Where loops don’t fit is clear too: when you can’t define done. “Keep going until this design is good” isn’t a goal.

Where I disagree or correct the original

1. “The first model where all three pay off.” The post ties the whole stack to one model release. My experience points the other way: the rules file, the checks and the loop decide the result more than the model does, and they worked with earlier models too. A better model makes the stack cheaper and lets it run longer, that’s all. Tying the stack to a model pushes you to rebuild everything at each release.

2. “If the six counters read zero, it’s done.” Checks only catch what they can measure, and sometimes measure the wrong thing. It happened to us twice in our own video production:

• In the voice-over every measure was clean: pitch, loudness, the tone-break test all passed. Still, watching it, I heard the voice change at the start of the last line. The cause was found later: the voice model pitched the end of a long script noticeably higher, and our threshold had let it through as a “natural rise.”
• The mouth-movement check was meant to confirm the mouth stays closed while the other speaker talks. It failed, and on inspection it was measuring head movement more than the mouth: it gave the same reading on the forehead, where there is no mouth.

The lesson: test the sensor too. When a check passes or fails, ask once what the number really measures. The final call belongs to human eyes and ears.

3. The model grading its own frames. The author counts this as an inferential sensor, yet the same post rightly says later “don’t let the agent that did the work decide the work is good.” The two contradict each other. What works for us is a separate agent that didn’t write the work: the two false claims in our script were caught by a separate reviewer, not by the one that drafted it.

4. Running everything on one strong model. The post recommends effort levels (medium for small fixes, high for new work). That isn’t enough. One of our videos ran every stage in a single session on the strongest model, and the session’s context grew to 815 thousand tokens. Even while waiting for a render or an avatar, every step re-processed that whole history. Most of the cost comes from accumulated context, not the model price. Our rule now: mechanical steps like running commands, waiting and converting go to a smaller model in a separate context; writing, diagnosis and risk decisions stay with the strong one.

5. Benchmark tables. The scores and prices in the post largely match Anthropic’s Opus 5.5 announcement; cache reads really did drop from $0.50 to $0.20 per million tokens. But Anthropic itself writes that at this level score gaps are a weak guide to real-world differences. Decide by measuring on your own work, not from a table.

6. Small sourcing errors. “Mini-SWE-Agent is about 100 lines” isn’t in the paper’s abstract; the 90-line minimal scaffold there is the paper’s own proposal. “Multi-agent setups use 15 times the tokens of a chat” doesn’t appear in the paper either. I left both out.

A sequence you can copy from our setup

1. Text first. No visuals until the text is final in both languages and its length is measured to fit.
2. Guides. One rules file (do’s, don’ts, the definition of done) and one skill file per recurring job.
3. Sensors. One computational check on every change and one inferential check by a separate agent. Verify once by hand what each check actually measures.
4. A goal. A machine-checkable stop condition, a turn limit, a stop-if-no-progress rule.
5. A record. Every failure becomes a lesson with its cause and permanent fix, and the next job starts from it.

The line all the sources share: when the same problem shows up twice, don’t fix the output, fix the harness.

Frequently Asked Questions

Can AI generate video files directly?

Not in the method described here. The language model does not output a video file; it writes a program that draws frames, and the video is produced by running that program. Generative video models that synthesize realistic footage are a separate category of tool.

What is the difference between a harness and a loop?

A harness makes a single run more reliable through rules, tools and checks. A loop runs that process repeatedly without a human prompting every step and keeps going until it meets a defined goal.

If all checks pass, is the work finished?

No. Checks only catch what they can measure, and sometimes measure the wrong thing. You should verify what each check actually measures at least once; the final decision belongs to a human.

Which steps should go to a smaller model?

Mechanical steps such as running a defined command and reporting the result, waiting on a long job, or converting files. Writing copy, finding the root cause of a failure and making irreversible decisions should stay with a stronger model.