Route slow media tasks away from short API tasks

A long video encode can occupy a worker while a short status-refresh job waits behind it. Put workloads with different resource needs on separate queues and assign workers explicitly to those queues.

Task routing assigns short work and media work to separate queues and explicitly assigned workers.

Route the actual registered task names

This toy Celery configuration assumes the named tasks are registered in the fixture.tasks module.

app.conf.task_default_queue = "short"
app.conf.task_routes = {
    "fixture.tasks.encode_video": {"queue": "media"},
    "fixture.tasks.refresh_status": {"queue": "short"},
}
app.conf.worker_prefetch_multiplier = 1

Start separate workers with separate names:

celery -A fixture worker -n short@%h -Q short -c 2
celery -A fixture worker -n media@%h -Q media -c 1

These example concurrency values make the local behavior easy to observe. They are not capacity recommendations for an unknown server. A media encoder can create its own threads or subprocesses, so one Celery slot does not necessarily mean one CPU core.

The default queue handles tasks without a matching route. Inventory those tasks and make sure a worker consumes that queue; an unconsumed queue can accumulate work while the rest of the deployment looks healthy.

Verify routing independently from throughput

Create a harmless long task that sleeps and a short task that returns a marker. Submit both and inspect worker logs to see which worker accepts each. Use synthetic task IDs and avoid real customer messages or media uploads.

The expected observation is placement on the intended worker, not a promised response-time improvement. Measuring latency needs a declared arrival pattern, worker resources and enough samples to show what happens under contention.

Prefetch changes how much work a worker reserves before execution. Reducing its multiplier can improve fairness for long tasks, but acknowledgement settings also affect reservations and redelivery. Do not enable late acknowledgement solely to obtain a particular prefetch behavior without making the tasks duplicate-safe.

A queue boundary does not isolate shared dependencies. Both workers may still contend for the database, disk, Redis or the provider quota. If media jobs saturate disk throughput, separating Celery queues alone will not protect a short task that needs that same disk.

Monitor queue age as well as length. A queue containing a few abandoned old jobs can deserve attention even when its count is low.

ShortForge provides media-processing context; Postzyo provides queue context. Celery documents routing and prefetch tradeoffs.