Case Study · Content Operations / Media
A Multi-Agent Video Editing System That Cuts at Human Granularity
The client publishes short vertical edits of live performances. Each video used to cost weeks of manual cutting; now 15 MCP tools, eight specialized agents, and a chat interface render a music-synced 50-second edit in 2 to 5 minutes. This page shows the trade-offs, the metrics, and the failures.
Client-approved publication. All metrics labeled: based on Fancam internal data, Jun 2026. No client infrastructure disclosed.
TL;DR: for business leaders
- One video cost weeks of manual editing; the old pipeline cut at 18-second sections, a human editor at 1-second beats.
- We built a multi-agent video editing system: 15 MCP tools, eight orchestrated agents, a pipeline grown from 20 to 48 edit slots in 4 rounds and 25 fixes.
- It renders in 2 to 5 minutes; 9 of 11 target metrics are green, and the two remaining failures have documented causes and a plan.
Problem: an 18-second brain on a 1-second job
The implementation plan names the gap: "A human editor operates at roughly 1 second. The pipeline operated at roughly 18 seconds." (Jun 2026). A human cuts on kick patterns, on the groove entering, on the voice arriving; a section-based pipeline sees one block covering all of them.
By hand, each video means a person watching, marking, trimming, syncing, reviewing, while the queue never shrinks. The founder's measure: "What took us weeks of manual editing now runs in minutes." (Founder's statement, Jun 2026).
For your company: if your output depends on a human at a timeline, your ceiling is attention, not capacity.
System: production and frontier, separated on purpose
Production is what runs today: a chat interface, an agent runtime exposing 15 MCP tools, a clip analyzer on the client's GPU servers, and a renderer. The chain: get_event_info() → search_clips() → plan_composition_with_ai() → build_composition() → render_video(). "Make me a 30 second video" becomes a finished file in 2 to 5 minutes.
The engineering frontier is the pipeline, at version 16. Input: one MP3, 16 raw clips. The stages (Demucs, librosa, MusicSectionDetection, WhisperX, DeepSeek V4 Pro) are detailed in the stack table below; the pipeline then scores clips deterministically, assigns them under hard quotas, edits with four parallel LLM agents, each followed by a deterministic validator, reviews, renders.
Audio + 16 clips → music analysis → clip scoring → orchestrator (LLM + clamps)
→ editing room (4 LLM editors + validators) → review agent → renderRead the engineering record of how we build: /engineering
Production architecture
This diagram shows a system that runs in production, not a slide.
For your company: production is what you buy; the frontier is where the next version comes from, and we show you both.
Decisions: four trade-offs made on purpose
1. Validators over prompts.
The LLMs were fragile: the rhythm editor ignored the stutter rule, the orchestrator collapsed sections. Everything that can be programmatic MUST be programmatic. Stutter cuts went 0 to 20 when validateRhythm enforced the ceiling in code. Trade-off: less creative freedom, no broken rules.
For your company: reliability is a code decision, not a prompt decision.
2. Creative intent, not pixel instructions.
DeepSeek emits "buildup" at "high" intensity; the render layer owns the parameters. Trade-off: less prompt control, more stability.
For your company: the system survives model upgrades.
3. DeepSeek API for orchestration, Qwen local for analysis.
Heavy analysis (vision, transcription, stem separation) runs on the client's GPUs; direction runs on DeepSeek's API. Trade-off: one external dependency, the expensive bulk stays local.
For your company: cost lands where it belongs; analysis is mass, direction is rare.
4. The diversity gate.
The orchestrator once returned missing_sections: ["verse", "outro"] and the video collapsed to intro plus chorus. The gate guarantees one slot per section type and caps the intro at 25% of target. Trade-off: short targets lose the outro; shape wins.
For your company: a structural guarantee beats a well-worded request.
Metrics: v7 to v16, with definitions
"Stutter cut" means a cut of 400ms or less inside an accent burst zone (kick density of 6 or more within a 2 second window). Total slots are the edit slots in the final timeline. Section types are the distinct musical sections represented. Editor failures are LLM agents that failed and required a fallback path.
| Metric | v7 (baseline) | v12 (seq fix) | v15 (8 fixes) | v16 (current) | Target |
|---|---|---|---|---|---|
| Section types | 2 | 4 | 4 | 3 (no outro) | ≥3 |
| Intro slots | 1 | 1 | 1 | 3 | ≥3 |
| Total slots | 20 | 25 | 28 | 48 | ≥35 |
| Total duration | ~42s | 36.1s | 43.5s | 47.2s | 50s |
| Stutter cuts | 0 | 15 (60%) | 14 (50%) | 20 (42%) | ≥8 |
| Clips used | 11/16 | 15/16 | 16/16 | 16/16 | 16/16 |
| clip_9 (dancing) | never | never | once, 383ms | chorus (artist_dancing) | at climax |
| clip_12 (dancing) | never | never | 2 times | intro (3rd slot) | at climax |
| best_section source | Phase4 (broken) | 50% intro forced | 19 Phase4 + 1 fallback | 20 Phase4 + 0 fallback | Phase4 preserved |
| Editor failures | Fallbacks | Fallbacks | 0 | 0 | 0 |
| Video size | 120.8 MB | 94.9 MB | 112.3 MB | 103.8 MB | - |
| Outro in 50s target | Yes | Yes | No | No | No |
Source: based on Fancam internal data, Jun 2026. Stutter is an absolute with its denominator: 20 of the 48 slots (42%). Of the 11 rows with targets, 9 are green; the two shortfalls (47.2s vs 50s, clip_12 in the intro) are in the audit below.
For your company: metrics with definitions are the difference between a demo and a system.
Honest assessment: the v16 audit
We scored v16 against the ideal editorial map: right clip, right section, right duration, right effects. Four of six moments missed; against v7, v16 crushes it. Both are true.
| Moment | Verdict | Root cause | Class |
|---|---|---|---|
| 0-14s intro | Partial 40% | Pool has no venue or atmosphere footage; dancing clips score 0.3 in the intro instead of 0 | Pool + scoring |
| 14-16s kick burst | Missed 0% | key_moments never reaches the orchestrator, so the stutter override cannot fire | Data (blocker 1) |
| 16-31s groove | Missed 0% | The rhythm editor ignores rhythmic_accents_ms; no zone-based pacing exists | Programmatic (blocker 2) |
| 31-32s burst | Missed 0% | Same data gap: key_moments is absent | Data (blocker 1) |
| 33s climax | Missed 0% | No key_moments, no vocal_entry override, no hold; the first clip lasts 383ms | Data (blocker 1) |
| 33-50s chorus | Partial 50% | The color validator cools to 5985K against a 5300K target | Programmatic (parameter) |
What each class means for your company:
- Pool: content gaps are a planning problem, not a code problem.
- Data: a ~10 line fix, not a redesign.
- Programmatic: pacing belongs in code; in prompts it drifts.
Open blockers: (1) key_moments never persisted to the orchestrator (data, ~10 lines); (2) no zone-based pacing in validateRhythm (programmatic, ~40 lines); (3) no venue clips in the pool (content); (4) dancing clips weigh 0.3 in the intro, not 0 (scoring).
Plan v17, documented next steps: propagate key_moments (~10 lines), add programmatic zone pacing (~40 lines), document the pool limitation, re-source venue footage.
For your company: an audit that names root causes and line estimates is a roadmap, not a confession.
This is the kind of audit we deliver: measured, labeled, with causes. One week, one process, $1,500.
1 week · You own the findings · 100% credited toward your build (within 60 days)
Stack: public parts, one system you own
| Component | Role | Layer | Where it runs |
|---|---|---|---|
| Demucs (Hybrid Transformer) | Stem separation: vocals, drums, bass, other | Deterministic neural model | Client GPU |
| librosa | Beat grid, kick grid (746 kicks), accent bursts, BPM | Deterministic | Client GPU |
| MusicSectionDetection | Section labels (Intro, Verse, Chorus, Bridge, Outro) | Deterministic model | Client GPU |
| WhisperX | Word-level transcription for the lyrics timeline | Deterministic model | Client GPU |
| DeepSeek V4 Pro (API) | Music enrichment, orchestration, 4 editing agents, review | LLM | API |
| Qwen3-Omni (local) | Clip vision analysis: narrative labels, action, faces | LLM | Client GPU |
| Remotion + Chromium | 30fps MP4 render | Deterministic | Client GPU |
| ComfyUI | Experimental frame generation for AI clips | Diffusion (not LLM) | Client GPU |
The production chat layer exposes 15 MCP tools: get_event_info, search_clips, get_thumbnail, plan_composition_with_ai, build_composition, start_preview, render_video, upload_image, the pattern tools load_patterns, save_pattern, get_artist, plus tools for media metadata, job status and session handling.
For your company: every component is public software; the orchestration is the moat, and you own it.
Prompts: three instructions that carried the project
RULE PRECEDENCE: P1 (quotas) → P2 (variety) → P3 (must_include) →
P4 (diversity) → P5 (compatibility). Higher priority ALWAYS wins.
If P4 conflicts with P2, P2 wins. No exceptions.Why: the orchestrator's worst failures were rule conflicts the model could not arbitrate. Priority removed the ambiguity; code enforces the ceilings.
P2: VISUAL VARIETY (THE #1 FAILURE MODE):
no single source clip may occupy more than 40% of any section's slots.
A flagged gap is 10x better than a loop.Why: one clip reused 9 times created a visually broken loop. The rule prices an admitted gap below a repetition.
Do NOT generate pixel-level instructions.
Emit creative intent: {"effect_intent": "buildup", "intensity": "high"}.
Specialized tools translate intent into concrete render parameters.Why: separation of responsibilities. The director decides the feeling; the tool owns the rendering.
Three quotes that sum the judgment:
"Everything that can be programmatic MUST be programmatic."
Reliability doctrine.
"DeepSeek should not generate pixel-level instructions. It should generate creative intent that specialized tools interpret."
Architecture doctrine.
"The validator frames, it doesn't override."
The 50/50 blend: validator value and programmatic target, half and half.
For your company: prompts are contracts, not magic. The code underneath is what you can audit.
Lessons: 4 rounds, 25 fixes, one doctrine
Round 1 (v7 to v9): diversity gate, deterministic best_section (the vision model hallucinated, giving 47% of clips best_section=intro), stutter made programmatic. Round 2: one retry with feedback, zero fallbacks. Round 3: preserve validated values, 10-beat cap, all 16 clips used. Round 4: intro capped at 25%, minimum 3 intro slots, key moment overrides, hard-blocked clip preferences.
The doctrine (project state document, Jun 2026): "Each programmatic fix is 10x more reliable than an LLM prompt. The forced stutter in validateRhythm() works. The stutter via the rhythm editor prompt does not."
Process context (session logs, Jun 2026): 408 sessions, 11,704 messages, 24 days. Validators: validateRhythm (durations, stutter clamps), validateVisual (repetition, transitions), validateColor (temperature arc, 50/50 blend), applyKeyMomentOverrides (vocal entries, climaxes), computeBestSectionDeterministic (clip matching), diversity gate (coverage).
For your company: the doctrine transfers; the deterministic parts of your workflow belong in code, not in requests.
Start with evidence, not a pitch
Start with an Audit ($1,500)
One week, one critical process, and you get the same measured verdict applied to your workflow. You own the findings; it is 100% credited toward your build within 60 days.
1 week · You own the findings · 100% credited toward your build (within 60 days)
Book a Strategy Call
30 minutes, no pitch deck; we will tell you where agents help and where they do not.
Skip to Custom System ($15K+)
Clear scope and budget? Custom systems from $15K; every line of code is yours.
Measured verdicts, not slides. No invented numbers here, and none in your audit either.
FAQ
Why not build this in-house?
The gap between "LLM calls that mostly work" and "0 editor failures" is the 25 fixes above: validators, gates, retries, clamps. The doctrine took 24 days of evidence to learn; skip the tuition.
Is $25K expensive?
Measured against weeks of manual editing per video, no. Weeks per video before, minutes per render now, in production. The $1,500 audit tells you what to build and is credited to the build.
Does it work with any genre?
The system reads numeric features, not genre labels: kick density, drum RMS, vocal RMS, energy curves. The test track was a reggaeton track at 117.45 BPM; the same 20 music-driven rules apply to rock, salsa and ballads.
What happens if it fails?
We do not build until the audit shows a case. Delivery is phased; the first agent is in production before the full budget is committed. You own the code, so a partial system is still yours.
Do I own the system?
Everything: code, prompts, architecture, model configuration. No licenses, no platform lock-in, deployed to your infrastructure. The client opened the code after the project ended; with us, that is the norm, not the exception.