AI Video Model Benchmark: 60 Attempts, a 12× Credit Spread
Quick answer
In this fixed 60-attempt benchmark, median charges per completed output ranged from 384 to 4,608 account credits, a 12× spread. All requests specified eight seconds, 720p and audio disabled. Fifty-eight completed; two returned a policy-coded rejection. Returned dimensions and frame rates differed. These are historical charges through Magic Hour, not quality rankings or current USD prices.
The same requested video settings produced a 12× spread in median completed-output credit charges across five model selections in this historical run. That does not mean the outputs were interchangeable: resolution details, frame rate and capabilities differed, and no human acceptance ratings were collected.
Cloud shadows move slowly across the scene while foreground plants respond to a light breeze. The camera makes one gentle push forward. Preserve the original composition, subjects, and visual style. One continuous shot.
Test an image-to-video brief
Upload a source image to Magic Hour, choose a current model, and review the entire clip against the details your project must preserve.
Try Image-to-VideoWhat was compared?
The requested models were Kling 3.0, Seedance 2.5, Veo 3.1, Sora 2, and LTX 2.5, all accessed through Magic Hour's image-to-video API. Each received the same source image and motion prompt for a skincare bottle, sneaker, coffee scene, and furnished room, with three attempts per model-scenario pair.
Settings were eight seconds, requested 720p, and audio disabled. The dataset was published September 9, 2026, with request timestamps on September 10 UTC. Magic Hour designed and funded the study and provides the API used for every model. This is one sponsored run through one integration, not an independent comparison of the providers' own applications.
Recorded completion and credits
Figure 1. Median charged account credits per completed output, September 9, 2026 Pacific. Twelve attempts per model across four original commercial scenes. Bars start at zero. Not direct-provider USD rates, current prices or cost per accepted output.
Requested model | Completed / attempted | Median credits per completed output |
|---|---|---|
12 / 12 | 384 | |
12 / 12 | 384 | |
12 / 12 | 768 | |
12 / 12 | 960 | |
10 / 12 | 4,608 |
The two Seedance ERROR records carried the nsfw policy code and zero final credits. The code does not establish actual prohibited content, a technical malfunction or a false positive. API completion was 58/60, or 96.7%, in this run; it is not a 96.7% usable-video or customer-success rate.
What the Completed Files Actually Returned
Requested Model | Returned Pixels / fps | Credits Across All 12 Attempts |
|---|---|---|
kling-3.0 | 1276x720 / 24 fps | 4,608 |
ltx-2.5 | 1278x722 / 24 fps | 4,608 |
veo3.1 | 1280x720 / 24 fps | 9,216 |
sora-2 | 1280x720 / 30 fps | 11,520 |
seedance-2.5 | 1276x720 / 24 fps | 46,080 |
All 60 attempts charged 76,032 credits in total. Credits are observed charges for this account and run. The completed-output medians omit rejected attempts; total charges retain them. The different returned pixels and fps are why requested 720p should not be described as identical returned specifications.
Use the versioned dataset and DOI for citation and the run-level results to inspect each request. Availability, routing, billing, and model behavior can change.
Inspect the actual commercial examples
The skincare contact sheet below shows the original source and all fifteen requested attempts for that scenario, including the failure. It contains still frames, so it cannot establish motion continuity, label stability across time, or overall clip quality.
Watch the complete output gallery before drawing a visual conclusion. The four scenarios answer different practical questions:
- Skincare: does a small camera orbit preserve the bottle, label symbol, liquid level, and pedestal?
- Footwear: do the shoe silhouette, laces, stitching, color blocks, and sole survive a controlled camera move?
- Coffee: can steam move while the bag, cup, beans, and counter remain consistent?
- Interior: do furniture identities, straight lines, room layout, and materials survive a dolly move?
These are review questions, not completed scores. Read the frozen study design and prompts to see exactly what each request asked for.
What can this benchmark establish?
It establishes the recorded completion status, charged credits, and downloadable outputs for these attempts. It also documents a shared request schema across the five named model selections through Magic Hour.
It does not establish a best-looking model, a cost per accepted ad, or a general reliability ranking. There were only twelve attempts per model, and no completed human quality scorecard. The collector checked jobs sequentially, so observed terminal time is an upper bound; it should not be presented as exact generation latency or used to rank model speed.
Sora's inclusion is historical access evidence for this run. OpenAI's shutdown notice lists September 24, 2026 as its API discontinuation date, after closing the consumer experience in April. Check current provider access before planning a new long-term workflow.
Use the evidence for your own product brief
Write the details that must survive before generating: product shape, readable label, material, included items, camera movement, and intended export. Choose one of the eight product-video prompt recipes and adapt it to a source image you have permission to use.
Start in Magic Hour Image-to-Video or use the API documentation for integration. Record the exact model and settings and retain failed and rejected attempts. Before calling an output usable, watch the whole clip against the source image and your written brief. Check these five things:
- Motion: does the camera or subject move as requested, rather than simply producing smooth but different movement?
- Product identity: do shape, proportions, materials and required objects stay consistent from start to finish?
- Labels and details: do the original symbols, colors and product details survive, without added branding or changed text?
- Continuity: is the clip free of distracting flicker, warping, duplicated objects or implausible movement?
- Export fit: does the complete file play, with the duration, orientation, resolution and audio your deliverable requires?
Write pass, minor defect or reject beside each check and explain any rejection. These are checks for your own brief, not completed scores for this dataset. A finished generation can still fail the brief; count accepted clips only after that review.
Use the video-cost worksheet to calculate total relevant spend divided by accepted outputs. Until the clips have been reviewed, this dataset supports cost per completed output, not cost per usable commercial deliverable.
Download a Reproducible Model Summary
Download the five-row summary CSV and read the fixed data card and derivation script. The script fetches the original JSONL pinned to Git commit 7bbd565, then recomputes completion counts, policy-coded rejections, total charges, medians and observed output metadata.
This adds a reusable summary, not a new experiment. The study remains v1.0.0 from September 2026. To rerun it, preserve the published scenarios and requested settings, verify current model access, record every attempted request, and release a new dated dataset with media checksums. Do not overwrite the historical edition or treat its sequential polling times as model-speed measurements.
Citation and reuse
Cite: Magic Hour AI, Inc., Commercial Image-to-Video Model Benchmark via Magic Hour, version 1.0.0, September 9, 2026, DOI 10.5281/zenodo.22683989. Include the sponsor, settings, sample size, and distinction between completion and quality when quoting results.
The repository release includes prompts, source images, outputs, checksums, and reproduction code. Methodology and result metadata use CC BY 4.0; code uses MIT; generated media have separate reuse terms.
For a broader platform decision alongside these controlled results, see the best AI video generators guide.
For a separate retained advertising-output evaluation, read the Fable 5 vs Sol 5.6 taste test.
Continue learning
Continue learning: use the complete AI video workflow to turn a benchmark result into a production process, then use the prompt troubleshooting guide when a clip fails.
Frequently asked questions
The requested models were Kling 3.0, Seedance 2.5, Veo 3.1, Sora 2, and LTX 2.5, all accessed through Magic Hour's image-to-video API. Each received the same source image and motion prompt for a skincare bottle, sneaker, coffee scene, and furnished room, with three attempts per model-scenario pair.
Settings were eight seconds, requested 720p, and audio disabled. The dataset was published September 9, 2026, with request timestamps on September 10 UTC. Magic Hour designed and funded the study and provides the API used for every model. This is one sponsored run through one integration, not an independent comparison of the providers' own applications.
It establishes the recorded completion status, charged credits, and downloadable outputs for these attempts. It also documents a shared request schema across the five named model selections through Magic Hour.
It does not establish a best-looking model, a cost per accepted ad, or a general reliability ranking. There were only twelve attempts per model, and no completed human quality scorecard. The collector checked jobs sequentially, so observed terminal time is an upper bound; it should not be presented as exact generation latency or used to rank model speed.
