Batch image and video generation without queue collapse

Illustration: separate image and video queues let each workload advance at its own pace.

A batch generator usually fails at the least interesting moment: not while generating one image, but when a customer imports 20,000 rows and the application turns all of them into live requests at once.

The API may be healthy and every payload valid. The system still collapses: database connections fill, callbacks arrive faster than you can commit them, one large tenant blocks everyone else, a retry duplicates an expensive video, and a traffic spike becomes a bill.

Keep waiting work in a durable queue, cap tasks in flight, and retry only when you know whether a remote task already exists. A larger Promise.all does none of those things.

One fact shapes everything below: SpicyAPI has no bulk or batch endpoint. Each row is one createTask call with its own task, hold and settlement, and an accepted task cannot be cancelled. The queue, the concurrency cap and the stop button therefore all live in your application.

Try it in the Playground first

Open the z-image Playground. Write a representative Prompt, choose the Aspect ratio, set Prompt expansion the way you plan to run the batch, press Run model and check the Output. Scale up only when that single result is what you want; a scheduler multiplies the request, including its mistakes.

The short answer

• Keep the backlog in your own durable queue, not in an HTTP request or an in-memory promise.
• Cap tasks in flight yourself. The platform's rate limit (1,000 requests per 10 seconds, separately for reads and writes) does not limit how many accepted tasks run at once; your balance and spend caps do. Pick a number, per media class, and hold it until the task is terminal.
• Admit by queue age, not queue length, and decide the cost of the whole batch before the first dispatch.
• One saved idempotency key per logical output, always sent with the same API key, and never trusted past 24 hours.
• Retry by error code: capacity and transport errors yes; invalid input, empty balance and spend caps no.
• Round-robin across tenants, and keep a dispatch kill switch that leaves completion handling running.

Three limits, and who sets each

Treat submission, dispatch and completion as separate stages:

client batch

│

▼

admission + cost check ──► durable queue ──► bounded workers ──► createTask

│

polling / webhook completion

│

copy output ──► result store + customer event

The rate limit governs submission, not concurrency: accepted tasks do not keep drawing from the request bucket while generating.

At 5 completions per second you create about 5 tasks per second, a small fraction of the 100 per second bucket. For most batches the submission limit is a safety fuse, not your throttle; the money held by in-flight tasks and your own caps stop you first. Polling uses the separate read bucket, so a runaway poll loop cannot block createTask, but it can start returning 429 on recordInfo and leave you blind. Read the header rather than hard-coding the defaults.

Other APIs can impose separate concurrency and pending-queue limits; Leonardo's concurrency guide distinguishes them. SpicyAPI has no documented numeric cap on tasks in flight, so configure your own rather than relying on its request-rate limit.

Size the in-flight cap from time and money

For one stable workload class, Little's Law relates the average number of tasks in the system to completion rate and the mean time each task spends there:

average tasks in flight ≈ completions per second × mean seconds from acceptance to terminal state

If image tasks average 8 seconds and you need 5 per second:

5 jobs/s × 8 s = 40 concurrent image tasks

Treat 40 as average occupancy, not a guarantee. A starting cap of 50 adds 25% headroom; raise it only while queue age, errors and spend stay healthy. Use p95 and p99 for tails and queue-age alarms. Measure task time from createdAt and completedAt in recordInfo, per model and settings.

Video belongs in its own pool. If Seedance 2.0 clips average 70 seconds in your workload and you want 0.2 per second:

0.2 jobs/s × 70 s = 14 concurrent video tasks

The 8- and 70-second averages are assumptions for sizing, not promised generation times.

Sharing one pool of 50, a burst of long videos can occupy every slot while quick images wait. With 50 image slots and 14 video slots both classes keep moving. Z-Image and Seedream 5.0 Pro can have separate pools too if their durations or priorities differ.

Now check the cap against money. Every accepted task holds its estimatedCost until it settles, and the hold counts toward every key and team cap. Using public catalog prices:

A balance that cannot cover those holds makes new tasks fail with 40201 no matter how good your scheduler is. Total spend matters more: 20,000 Seedream 5.0 Pro 1K images come to $720 and 20,000 Z-Image images at $0.01 with prompt expansion off to $200. Prices change, so read them from the live model page or a quote. The cap that stops a bad batch is the one you set before it starts: set a daily spend cap on the production key before dispatch, sized to the most you can afford to lose in a bad day. A blank cap means no limit; team limits can also stop new tasks with 40202.

The drain-time estimate is the other half:

drain time ≈ queued jobs ÷ sustainable completions per second

18,000 queued images at 5 per second is one hour of work. If you promise a 15-minute start time, admission has to slow down well before 18,000. A backlog that is growing is not steady state, so rely on observed queue age more than on the equations.

Backpressure starts before dispatch

Backpressure means the downstream system can make the producer slow down. Without it the queue absorbs work forever and the customer learns about the overload hours later.

Use two admission thresholds on queue age, the age of the oldest ready item. Length lies when items differ in cost; age says directly whether the promised wait is being met.

• Soft threshold: accept the batch, return a delayed estimate, slow the producer.
• Hard threshold: defer or reject new work with a retryable product response before any remote task exists.

Compare money in exact decimal units rather than binary floating point. Commit the reservation atomically with batch admission in your queue adapter, so concurrent submissions cannot both consume the same remaining budget.

Do not turn every internal overload into an API 429 for your own customers. If the work is durably stored, return an accepted batch ID in a delayed state. Reject only work you cannot responsibly keep, and when you do, give a retry time and make the batch submission itself idempotent.

Two details of this API decide how long an item may safely wait in your queue:

• Reference files expire in one day. A spicy:// URI from an upload is valid until the expiresAt in its commit response (one day by default), and a retry reuses the original input, so a retry after that returns 400. Keep a public HTTPS URL or your own source ID in the queue and upload at dispatch time.
• Quotes last five minutes. If you want a price lock, quote at dispatch and send quoteId with expectedCost to createTask; a changed price then fails with 40901 instead of charging a new amount. Quotes are free but count against the read bucket, so quoting 20,000 rows at admission is the wrong place.

Idempotency has two layers

Two different duplicates need preventing: the customer submitting the same batch twice, and your dispatcher not knowing whether a create succeeded. Give the batch a stable identity, then generate and save a UUID for each new logical item. Recovery reuses that item’s saved key.

Keep a unique constraint on that key in your database. Write the item as "intent" before calling the API, send the key as Idempotency-Key, and store the returned taskId on the same row. After a crash, the next worker reuses the saved key if acceptance is still unknown. Writing intent only after sending leaves a window in which a restart can buy the same output twice.

What the idempotency docs add, and the queue has to respect:

• A key is valid for 24 hours from the first accepted request; replays do not extend it. After that the same key creates a new, billed task. Before replaying an old queued or dead-lettered item, look up its stored taskId instead of trusting the key.
• The key belongs to the same account and the same API key, and to the exact model, input and callBackUrl. Changing anything returns 409. A sibling key on your account also gets 409, and cannot read the task: only the key that created a task can read or retry it, so use one key for create, poll and retry in a worker fleet.
• A replay returns the original task in its current state with no marker that it was a replay.
• A rare same-key race returns 409 "Idempotency key conflict, please retry shortly". Retry once after a few hundred milliseconds; the other two 409 messages will never succeed.

A dispatcher that holds the cap until the task ends

Save one object per line in batch.jsonl, for example {"model": "alibaba/z-image-turbo/text-to-image", "input": {"prompt": "<one row's prompt>"}}, using the chosen model's accepted inputs. Set SPICY_API_KEY, BATCH_BUDGET_USD and optionally BATCH_WORKERS (default 2). Run this Python example and approve the batch cap. It keeps each worker occupied until its task is terminal, settled and no longer preparing output.

Keep the batch-run folder for recovery and do not reorder its source rows. The same folder retains settled spending and reservations across restarts. Run only one coordinator against a folder.

import json
import os
import time
import uuid
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from decimal import Decimal
from pathlib import Path
from threading import Event, Lock

BASE = "https://api.spicyapi.ai/api/v1"
API_KEY = os.environ["SPICY_API_KEY"]
CAP = Decimal(os.environ["BATCH_BUDGET_USD"])
WORKERS = int(os.environ.get("BATCH_WORKERS", "2"))
ROOT = Path(os.environ.get("BATCH_DIR", "batch-run"))
ROOT.mkdir(exist_ok=True, mode=0o700)
lock, stop = Lock(), Event()
reserved = Decimal("0")


class APIError(Exception):
    def __init__(self, status, body):
        self.status, self.code = status, body.get("code")
        super().__init__(f"{status}/{self.code}: {body.get('msg')}")


def call(path, payload=None, key=None):
    headers = {"Authorization": "Bearer " + API_KEY, "Content-Type": "application/json"}
    if key:
        headers["Idempotency-Key"] = key
    for attempt in range(5):
        request = urllib.request.Request(BASE + path,
            data=None if payload is None else json.dumps(payload).encode(), headers=headers)
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                body = json.load(response)
                if body.get("code") != 200:
                    raise APIError(response.status, body)
                return body["data"]
        except urllib.error.HTTPError as error:
            try:
                body = json.load(error)
            except ValueError:
                body = {}
            if error.code != 429 and error.code < 500:
                raise APIError(error.code, body)
            delay = float(error.headers.get("Retry-After") or min(15, 2 ** attempt))
        except (urllib.error.URLError, TimeoutError, ValueError):
            delay = min(15, 2 ** attempt)
        time.sleep(delay)
    raise RuntimeError("Unresolved API call; preserve this row for recovery")


def save(path, row):
    temporary = path.with_suffix(".tmp")
    temporary.write_text(json.dumps(row))
    temporary.chmod(0o600)
    temporary.replace(path)


def worker(pair):
    global reserved
    index, line = pair
    if stop.is_set():
        return
    try:
        request = json.loads(line)
        if not isinstance(request.get("model"), str) or not isinstance(request.get("input"), dict):
            raise ValueError("model and input are required")
    except (ValueError, TypeError, AttributeError) as error:
        print(index, "invalid local row:", error)
        return
    path = ROOT / f"{index}.json"
    row = json.loads(path.read_text()) if path.exists() else {
        "request": request, "key": str(uuid.uuid4())}
    if row["request"] != request:
        stop.set()
        raise ValueError("Batch changed; preserve the old folder and use a new one")
    if row.get("settled") or row.get("rejected"):
        return
    accounted = "sentAt" in row
    try:
        if not row.get("taskId"):
            if time.time() - row.get("sentAt", time.time()) >= 23 * 3600:
                raise RuntimeError("Reconcile old acceptance before submitting again")
            if "submission" not in row:
                quote = call("/jobs/quote", request)
                row["hold"] = quote["maxCharge"]
                row["submission"] = {**request, "quoteId": quote["quoteId"],
                                     "expectedCost": quote["estimatedCost"]}
            with lock:
                if stop.is_set():
                    return
                hold = Decimal(row["hold"])
                if not accounted and reserved + hold > CAP:
                    stop.set()
                    print(index, "batch budget reached")
                    return
                if not accounted:
                    reserved += hold
                    accounted = True
            row.setdefault("sentAt", time.time())
            save(path, row)
            accepted = call("/jobs/createTask", row["submission"], row["key"])
            row["taskId"] = accepted["taskId"]
            save(path, row)
        task_id = row["taskId"]
        for _ in range(3000):
            task = call("/jobs/recordInfo?taskId=" + urllib.parse.quote(task_id))
            if task["state"] not in {"queued", "running", "succeeded", "failed", "expired", "canceled"}:
                raise RuntimeError("Unknown state: keep the reservation")
            if task["state"] in {"succeeded", "failed", "expired", "canceled"} and task.get("settled") is True:
                assets = task.get("output", {}).get("assets", [])
                if task["state"] == "succeeded" and any(a.get("pending") for a in assets):
                    time.sleep(5)
                    continue
                row["result"], row["settled"] = task, True
                save(path, row)
                with lock:
                    reserved += Decimal(task["cost"]) - Decimal(row["hold"])
                print(index, task_id, task["state"], task["cost"])
                return
            time.sleep(5)
        raise RuntimeError("Wait interrupted; resume the saved task, not a new generation")
    except APIError as error:
        if error.status == 400 and error.code in {400, 40003, 40004} and not row.get("taskId"):
            row["rejected"] = str(error)
            save(path, row)
            with lock:
                if accounted:
                    reserved -= Decimal(row.get("hold", "0"))
            print(index, "rejected input:", error)
            return
        stop.set()
        print(index, "paused:", error)
    except Exception as error:
        stop.set()
        print(index, "paused:", error)


if CAP <= 0 or not 1 <= WORKERS <= 20:
    raise ValueError("Use a positive budget and 1–20 workers")
# Count settled spend from this batch again on restart.
for path in ROOT.glob("*.json"):
    row = json.loads(path.read_text())
    if row.get("settled"):
        reserved += Decimal(row["result"]["cost"])
    elif row.get("sentAt") and not row.get("rejected"):
        reserved += Decimal(row["hold"])
if input(f"Authorize this batch up to ${CAP}? Type yes: ") != "yes":
    raise SystemExit("No tasks submitted")
rows = Path("batch.jsonl").read_text().splitlines()
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    list(pool.map(worker, enumerate(rows)))

The production integration documentation covers deployment. This example isolates invalid rows, pauses new admission for balance or uncertain errors, and bounds transient retries. Already accepted tasks keep being collected after admission pauses. It deliberately makes no automatic paid retry; a failed row remains recorded for an explicit new attempt after settlement. Run one such process per media class and put tenant round-robin in front of the file.

Fair scheduling: protect small tenants

First-in-first-out is fair to jobs, not customers. If tenant A enqueues 100,000 items a minute before tenant B enqueues ten, B waits behind all of A. Keep a queue per tenant and take one item per tenant per round:

round 1: A1, B1, C1

round 2: A2, B2, C2

Weights can reflect purchased capacity, but give every active tenant a minimum share. Also cap each tenant's in-flight count: the global cap protects your account, the tenant cap contains noisy neighbours. Inside a tenant, priority lanes (interactive previews, normal batches, backfills) are worth adding only once preview latency is a real complaint. If you do, age priority with queue time, otherwise a steady stream of previews starves a backfill forever. Everything on one account shares the same two request buckets and the same balance, so tenant fairness is your job, not the API's.

Classify failures before retrying

"Retry three times" is not a policy. The right action depends on what you know and whether a task may already exist. This table uses the documented codes; code values are in the response body, and task failures arrive as state: "failed" with an errorCode on a successful recordInfo call (code: 200).

Use randomized backoff so workers do not retry in lockstep.

Move confirmed failures and rejected inputs to a dead-letter queue with their classification and safe replay action. Keep an ambiguous create or unresolved accepted task in the reconciliation queue with its saved key and task ID; a local retry deadline does not prove it failed. A failed task normally releases its hold in full, but some chargeable content_rejected outcomes settle at the quote, so read settled and cost rather than assuming a refund.

Cost protection belongs in the scheduler

Concurrency limits protect machines; they do not protect money. A perfectly stable worker can still dispatch an unaffordable batch. Check the estimated cost against every envelope before dispatch:

allowed = tenant daily remaining

∩ tenant monthly remaining

∩ project batch remaining

∩ global hourly remaining

Reconcile against the real settlement: createTask returns the held estimatedCost, a successful task settles at or below it, and settled: true with cost is the final charge. Release your own reservation for items the API rejected, and keep the reservation for accepted ones until settled flips. Do not guess from an HTTP timeout.

Useful guards:

• a maximum estimated cost per batch, shown to a person before a large one starts;
• a per-tenant hourly dispatch budget and a global spend-velocity alarm;
• a circuit breaker per model or tenant;
• API-key caps set below the account balance, and a low-balance alert;
• a kill switch that stops new dispatch while leaving completion processing running.

Because accepted tasks cannot be cancelled, stopping creation is the only brake: the in-flight cap, not a cancel button, limits damage in a cost incident. Keep receiving webhooks, polling old tasks, settling reservations and copying outputs.

Completion at batch scale

The per-task callBackUrl delivery times out after 15 seconds; a registered account endpoint can receive its own delivery too, so the receiver should verify the signature against the raw body, write the event to an inbox table, return 2xx, and let a worker do the rest. Deduplicate on the v2 delivery request_id, and decide from data.state, not arrival order. Keep a slow poll as a fallback for missed events (see the webhook docs).

Copy outputs to your own storage as tasks finish. Result links last about 20 minutes and generated media is kept for about 14 days at most (retention); at 5 completions per second the copy step is part of your throughput, and a slow uploader becomes the bottleneck that hides in "completed" tasks. Shortening retention can also remove results for tasks you already ran.

Observe flow, not just errors

A useful dashboard answers four questions: are we accepting work, is it starting, is it finishing, is cost settling as expected? Track by model, media class, tenant tier and outcome:

Carry one correlation chain (batchId, item ID, idempotency key, taskId, request_id) through logs and events, and keep prompts, API keys and private asset URLs out of metric labels and routine logs. Alert on symptoms: "50,000 items queued" may be normal; "oldest interactive item is 12 minutes old" is actionable.

Start with a bounded batch, then increase capacity

Increase capacity only while oldest-item age, task failures and spend stay within your thresholds; more workers cannot fix an overloaded completion path.

Before the first paid batch, use a mock to interrupt a worker immediately before and after submission and to return 429, failed tasks and spend-cap errors. Recovery should reuse saved intent, keep unknown tasks unresolved, and stop new dispatch without stopping completion. Then run a small approved batch before opening admission to large imports.

For completion handling, use Polling vs webhooks for async generation jobs; for a batch budget, see Per-unit pricing vs subscriptions. The production checklist covers key configuration and launch tests.

FAQ

Does SpicyAPI have a batch endpoint or a concurrency limit?

No batch endpoint: each row is one createTask, and there is no documented cap on tasks in flight (limits are in the table above; current rate in X-RateLimit-Limit). If the rate limit is too low for your peak, ask console support for a higher one, describing peak requests per second and batch size.

Can I cancel a batch that was started by mistake?

No. Use the dispatch kill switch to stop new tasks while completion handling keeps running.

Should a 429 consume a retry attempt?

Wait Retry-After plus jitter and keep the same idempotency key. Repeated throttling calls for a lower dispatch rate. Bound retries by both attempts and time; if acceptance is still uncertain when that budget runs out, reconcile the saved intent before creating a replacement.

How do I stop duplicate charges after a timeout?

Re-send with the same Idempotency-Key and API key within 24 hours, and store the taskId as soon as you have it. After 24 hours, look up the stored taskId instead.

Is a queue enough to prevent overload?

No. A queue stores overload; backpressure controls it. You still need admission thresholds on queue age, an in-flight cap and a product answer for when the promised wait cannot be met.

What is a good starting concurrency?

Completions per second times mean task time, per media class, plus about 25% headroom: 40 for 5 images per second at 8 seconds each. Raise it only while queue age, errors and spend stay healthy.

Can high-priority jobs bypass the limits?

No. Priority chooses among admissible work and never bypasses the global cap or budgets. Give interactive work reserved capacity, and age lower-priority work so it is not starved.

Let an Agent handle the call

Use an Agent when you want it to prepare the inputs and return the result while you approve the price and creative choices.

• CLI: with one row's input object saved as input.json (for Z-Image: prompt, aspect_ratio, enable_prompt_expansion, output_format, seed), use npx @spicyapi/cli tasks create --model alibaba/z-image-turbo/text-to-image --input-file input.json --wait. The CLI confirms the price; keep the returned task ID for recovery.
• MCP: ask your assistant to use spicyapi_task_create for alibaba/z-image-turbo/text-to-image, then spicyapi_task_wait. The MCP server requests price confirmation before creation.
• Skill: install with npx skills add spicyapi-ai/spicy-skill, then ask $spicyapi to run the same workflow and stop for your approval. The Skill uses a configured CLI or MCP connection.