Every new video model ships with the same three seconds: a face turning toward camera in golden-hour light, skin texture immaculate, the background motion blurred just enough to look expensive. It's genuinely good. It also tells me nothing about whether that model can hold a talking head for eight seconds while the client's logo sits in the corner, or whether the hands survive a product close-up on the third revision.

I've switched models on the strength of clips like that. Twice it was a mistake, and the cost wasn't the subscription — it was a week of relearning syntax and rewriting prompts that already worked. So here's the argument I'd make to anyone currently being talked into a switch: the difference you see between two models in a demo reel has almost no predictive power for your project. Not because the demos are dishonest. Because they were produced under conditions you don't have.

## The demo is answering a different question

Every demo you watch was made with optimal input and no delivery constraints.

Optimal input means the reference frame was picked by someone who had already generated dozens of variants and knew which one holds up. The prompt was tuned over many runs, and the clip you see is the best of them. That's a legitimate way to show a model. It's also a completely different task from the one you're hiring the model to do.

No delivery constraints means nobody asked for a vertical cutdown. Nobody needed the shot to land at 6.5 seconds instead of 8. Nobody needed the same face to survive eleven consecutive clips in a sequence, and nobody had a client who wants the product label readable at second four.

The question a demo answers is "what is this model's ceiling?" Your question is "what is its yield?" Those are different numbers, and they don't rank models in the same order often enough for one to substitute for the other.

## The PK video is an even weaker signal

Official demos at least come from people who built the thing. Comparison videos are edited for watchability, which means the edit removes exactly the information you need. A side-by-side never shows you attempt four. What you want to know is how many tries it took to get the clip you're watching, and that number is the first thing on the cutting room floor.

There's also a structural reason the format can't help you: "both are fine, it depends on your shots" doesn't get clicked. That's not malice, it's selection. The genre is built to produce a winner, and your project may not be the one that winner was built for.

## What actually predicts your project: yield

For a given shot type and a given model, yield is attempts per usable output. If one model lands a usable take on attempt one and the other needs three, that's a threefold difference in the time cost of your most common shot — and across a twenty-shot project, that difference is your entire schedule.

Peak quality and yield are correlated, but not the same thing, and they invert often enough that you can't swap one for the other. What predicts your delivery is yield on the three shot types you shoot most. That number is computable in an afternoon, on your own footage.

## The test: three shot types, two models, two attempts each

### 1. Pick three shot types you actually shoot

Three, not one, and types rather than a single hero shot — one shot gets lucky, a shot type averages the luck out.

If your work looks anything like mine, the three are: a talking-head medium shot (face stability, lip sync, subtle motion), a hand close-up (where fine detail fails loudest — product handling, gestures, fingers near objects), and an environment transition (a camera move through a space with no subject to distract from warping). If half your output is product turntables or food pour shots, test those instead. The categories matter less than the rule: they should be the shots you generate most often, not the shots that look best.

### 2. Freeze the prompt skeleton before you run anything

Write the scaffold down and reuse it verbatim across both models. The skeleton I work from looks like this:

```
[shot type], [subject + wardrobe], [framing/lens], [camera move],
[lighting], [motion beat across N seconds], [style], [negatives]
```

The point isn't that this is the best skeleton in existence. The point is that it doesn't change between Model A and Model B — otherwise you're measuring your prompt writing, not the models. Adapt syntax only where a platform forces it (camera-move keywords, aspect-ratio flags), do that adaptation once, and then lock it. If you'd rather not build the scaffolds from scratch, there are ready-made ones for image, video, and music work at [awesome-prompts.com](https://awesome-prompts.com); I keep two of them in a notes file for exactly this purpose.

### 3. Two attempts each — and log the failures

Three shot types, two models, two attempts: twelve generations. Small enough to run while a client call happens in the background, large enough to separate "lucky" from "reliable-ish."

Two attempts is deliberate. One attempt can't distinguish luck from reliability, and five attempts starts measuring the ceiling — which is the number you already have from the demos.

The log matters more than the generations. For each of the twelve, write down three things:

- The attempt index where you got something usable — first, second, or never
- What specifically failed on the others: identity drift, hand melt, background warp, camera move ignored, duration cut short
- Whether it survived your real constraint — aspect ratio, target duration, continuity with the previous shot in the sequence

### 4. Score attempts-to-usable, weighted by your most common shot

Make a three-by-two grid. Rows are shot types, columns are models, cells say first / second / never.

Then weight rather than count. If most of your delivery is talking heads, the talking-head row decides the argument and the other two rows are tiebreakers. If your mix is even, add up the attempt indices and take the lower total. The output of this exercise is one sentence you can say out loud: "For my three most common shots, this model gets me something usable in about X attempts."

## The decision rules I use

Lower attempts-to-usable on my most frequent shot wins, even when the other model's best output is prettier. Deliveries are judged on the shots that came out, not the ones I imagined.

If the two models are within one attempt of each other across the grid, I don't switch. Relearning syntax, rewriting a prompt library, re-testing every template — that's a real cost and it never appears in the demo reel.

I don't switch because of an announcement. I switch when the grid says so, or when a brief requires a shot type neither model can produce.

A model that is merely good at everything but rarely fails beats a model with a spectacular ceiling and a one-in-five yield. The first one lets me quote a deadline.

## What breaks this test

Twelve generations can't prove anything statistically. Treat it as a filter, not a verdict — it's designed to eliminate an obviously wrong option in an hour instead of discovering the problem over a bad quarter.

Syntax mismatch is the other trap. If one model wants a genuinely different prompt dialect, give it one fair round of adaptation before you freeze the skeleton, or you'll end up measuring your own unfamiliarity.

Dirty input ruins it too. If your reference is a blurry screenshot, both models fail and you'll blame the wrong thing. Clean input moves yield more than most model switches do.

And the grid goes stale. If you tested three shot types this month and your briefs shift to six next month, re-run the test when your work changes shape — not when the news does.

## The questions that come up the night before a delivery

### What if neither model gives me a usable shot on the second attempt?

Suspect everything upstream of the model first: the reference frame, the aspect ratio, the duration. A shot that needs nine seconds of consistent motion often needs to be split into two generations and cut together. Before concluding that both models are bad, rerun the same prompt with a cleaner reference and a shorter duration. Duration is the cheapest variable you can change.

### Should I test with a reference image or text only?

Test with the input you'll actually use in production. If your workflow starts from a generated still, test from a still — text-only results will look more impressive and mean less. If you generate those stills in another tool, how good that tool is is a separate question from which video model to buy.

### How often should I re-run it?

When a shot type becomes common in your work, when you're about to commit to an annual plan, or when a wrong choice would cost you a reshoot. That's it. Not every release.

The next time a clip makes you want to switch, save it — but don't act on it. A demo reel is a claim about a ceiling. The only thing that predicts your project is what happens on your own footage, on your three most common shots, at an attempt count that fits your budget. Pick those three shots today, before the next announcement lands. The answer is usually less dramatic than the feed, and considerably cheaper than finding out mid-delivery.