Design a Celery task that can run twice safely

A queued task can be delivered more than once. If every delivery immediately sends a message, a retry can produce duplicate customer-visible work. Separate the intended action from the task delivery and give it a durable state.

Duplicate tasks claim one pending row; a lost provider response requires reconciliation before another send.

Claim the action before calling the provider

This illustrative Django/Celery fragment assumes a Delivery row with a unique provider_key, a state field and a provider_id. The provider adapter is application code.

from celery import shared_task

@shared_task
def send_delivery(delivery_id):
    claimed = Delivery.objects.filter(
        pk=delivery_id, state="pending"
    ).update(state="sending")
    if not claimed:
        return "already claimed"

    delivery = Delivery.objects.get(pk=delivery_id)
    try:
        receipt = provider.send(
            payload=delivery.payload,
            idempotency_key=delivery.provider_key,
        )
    except TimeoutError:
        Delivery.objects.filter(
            pk=delivery_id, state="sending"
        ).update(state="uncertain")
        return "needs reconciliation"

    Delivery.objects.filter(
        pk=delivery_id, state="sending"
    ).update(state="sent", provider_id=receipt.id)
    return "sent"

Track the age of unresolved records and provide an operator action that records the reconciliation evidence. An uncertain delivery should never disappear into an ordinary success count. The conditional database update stops concurrent task deliveries from both claiming the same pending row. The provider key must be stable for the intended action, not regenerated for each retry.

Provider-side idempotency is a separate contract. Some APIs do not support it, and supported keys may have retention limits or payload restrictions. This adapter signature is a demonstration of the boundary, not a claim that every messaging API accepts such an argument.

Recover the ambiguous states

A process can die after the provider accepts the message but before the local row becomes sent. A database claim alone cannot establish exactly-once external execution. A row left in sending should enter reconciliation, with a readback or other provider-specific investigation before another send.

The example catches only a timeout. A complete implementation also classifies definitive rejection, authentication failure and unexpected exceptions. Never reset all sending rows to pending just because they are old; a slow worker may still be active.

Test duplicate task calls with a stub that counts sends, a provider rejection and a provider that applies the action then times out. The last case is the one most likely to reveal an unsafe automatic retry policy.

The publishing workflows in Postzyo provide related context. Celery's task guide discusses duplicate-safe work and acknowledgement choices.