All stories
VibeVideoBench Ecosystem Community reports

AI video agents keep missing the final render

VibeVideoBench researchers say late, slow or never-started final renders explain 169 of 224 runs that failed to deliver a playable video.

By AITrending2 min read
Sources (4)

Why now The paper was revised on October 27 at 16:22 UTC and appeared in an October 28 Hugging Face community submission. Its first arXiv submission was October 26, not today's discovery date.
English VibeVideoBench diagram shows video tasks, rubric construction and a one-hour agent rollout with delivery and quality evaluation.
Author-published benchmark design diagram from Figure 2. The evaluation setup and findings are the research team's, not independently reproduced by AITrending. Source: Muxi Diao et al. / VibeVideoBench

Writing the animation code is not the same as delivering the video. VibeVideoBench researchers report that 224 runs in their coding-agent evaluation produced no playable final video. In 169 of those runs, the final render started too late, took too long or never started at all.

That makes the benchmark useful beyond another model leaderboard: it examines the point where a working creative prototype has to become an actual file someone can watch.

A one-hour production deadline

The authors tested nine complete agent-and-harness systems on 239 tasks across 12 categories, producing 2,151 scored rollouts. Each run received a fresh Docker sandbox with eight CPUs, 32 GB of RAM, no GPU and a 60-minute wall-clock limit. Agents could use tools including Python, Node, a headless browser and ffmpeg. Image- and video-generation models were excluded; text-to-speech was provided.

The task was to deliver a playable MP4 and supporting editable project material under the brief’s requirements. The evaluation combines programmatic delivery checks, task-specific rubric checks and judged video quality. These are comparisons of complete deployed systems, including their native harnesses, rather than isolated model weights.

The render pipeline becomes the bottleneck

The paper identifies 65 runs that never began their final render. It also describes 33 deadlocks caused by feeding frames to ffmpeg without draining its error-output stream, and 12 background renders cut off after the agent ended its turn.

The authors’ practical lesson is to finish a complete version early, then improve copies while preserving a deliverable. A preview or successful code edit does not protect a run from a blocked renderer or an expired deadline.

Delivery and quality also differ. Opus 5.5 with Claude Code delivered videos on all 239 tasks and led the paper’s strict rubric-success measure at 76.3%; Astra with Codex delivered 238 and led overall judged quality. The strict measure requires no failed core check and at least one passed core check; unknown and not-applicable checks do not count against it. It is not proof that every requirement was established.

Research results, with release limits

The paper first reached arXiv on October 26 at 17:59 UTC and was revised on October 27 at 16:22 UTC. An October 28 Hugging Face submission provides a fresh community discussion point, not a new original publication date.

These findings are the research team’s reported results; AITrending has not reproduced the runs. The project’s code and dataset links were still marked coming soon when reviewed. The benchmark’s bounded tasks, hardware and deadline also limit how far its rankings can be generalized to other video workflows.

Related reading