Separate retryable failures from permanent Celery errors

A temporary connection failure and an invalid recipient should not follow the same retry policy. Repeating a permanent failure wastes worker time and can bury an actionable error under several identical attempts.

Error branches leading to retry, review or failure.

Retry a narrow exception class

Keep retry attempts distinguishable from the intended business action in the audit log. The following task fragment uses an application-defined provider adapter. Its adapter must classify errors according to the actual provider's response contract.

from celery import shared_task

class RetryableProviderError(Exception):
    pass

class PermanentProviderError(Exception):
    pass

@shared_task(
    autoretry_for=(RetryableProviderError,),
    retry_backoff=2,
    retry_backoff_max=120,
    retry_jitter=True,
    max_retries=4,
)
def refresh_snapshot(account_id):
    account = Account.objects.get(pk=account_id)
    if not account.external_id:
        raise PermanentProviderError("Account is not connected")
    return provider.fetch_snapshot(
        account.external_id,
        connect_timeout=5,
        read_timeout=20,
    )

The timeout names above describe the adapter's interface, not a universal HTTP client API. The adapter should wrap a known retryable outage in RetryableProviderError while preserving an internal diagnostic.

A missing connection is an explicit permanent failure for this task invocation. The user may fix the account and submit new work later; that does not mean a worker should keep repeating the unchanged invalid request now.

The numeric delay limits are example settings. Choose them from the workflow's useful lifetime and provider policy, then declare them in configuration.

Include the final failed attempt

After the retry limit, persist a terminal outcome the operator can inspect. "Retry scheduled" is not a successful refresh, and a task stuck in retry should not cause the UI to show a current snapshot timestamp.

A 429 response may carry a Retry-After instruction. A generic exponential schedule does not automatically interpret every provider header, so build that behavior intentionally when required. Authentication errors may require token refresh or user reconnection instead of an ordinary retry.

Use a stub that returns a transient failure twice and then succeeds. Confirm attempts are bounded and the final data is recorded once. Use another stub that raises a permanent error and confirm it is not retried. Exercise exhaustion as its own case.

Retries must still respect rate limits and duplicate-safe processing. Backoff reduces repeat frequency; it does not undo an external action whose response was lost.

Postzyo provides related snapshot and queue context. Celery documents automatic retry and backoff options.