Reconcile scheduled work from durable application state

A scheduler tick can be missed, a broker can reject a message, or a process can stop after committing an intended action. Keep the business schedule in durable application state and periodically look for work that still needs an attempt.

Reconciliation revisits pending due records; duplicate enqueue attempts still require a duplicate-safe worker.

Revisit due records in bounded batches

This illustrative Celery task assumes a toy ScheduledPost model. Its publish_one task must claim each pending record atomically and handle duplicate deliveries safely.

import logging
from celery import shared_task
from django.utils import timezone

logger = logging.getLogger(__name__)

@shared_task
def reconcile_due_posts():
    due_ids = list(
        ScheduledPost.objects.filter(
            state="pending",
            due_at__lte=timezone.now(),
            attempts__lt=5,
        ).order_by("due_at", "pk")
        .values_list("pk", flat=True)[:100]
    )
    enqueued = 0
    for post_id in due_ids:
        try:
            publish_one.delay(post_id)
        except BrokerUnavailable:
            logger.warning("enqueue unavailable post_id=%s", post_id)
            break
        enqueued += 1
    return {"enqueue_attempts_accepted": enqueued}

BrokerUnavailable is a placeholder for the adapter's explicitly classified connection failure. Use the actual broker client's documented errors in a real implementation.

The result counts accepted enqueue attempts, not completed publications. A broker acknowledgement does not establish that a worker executed the action or that the provider accepted it.

Records stay pending until a worker claims them. Two reconciliation runs can therefore enqueue the same ID. This design relies on the worker's durable conditional claim and provider-outcome handling; merely assuming that scheduled tasks never overlap would leave a gap.

Separate overdue from uncertain

A record marked sending after a worker crash may require reconciliation with the provider. It should not automatically reenter this pending-only query. Keep uncertain external outcomes in a separate review path so the reconciler cannot turn a lost response into a duplicate post.

The attempts field also needs a precise meaning: business execution attempts, not every periodic enqueue. Update it consistently with the worker's state transition and record the final exhausted status.

Test a missed tick, broker failure on the second item, duplicate enqueue and a worker failure after its claim. Confirm later scans recover pending work while terminal and uncertain records remain outside automatic publication.

Track the age of the oldest eligible record. A successful reconciler invocation can still make no business progress if workers are stopped or the same records are repeatedly rejected.

The scheduling architecture in Postzyo provides related context. Celery documents periodic tasks and task execution behavior.