Trace a request across API and worker boundaries

A background failure is easier to investigate when its log can be connected to the request that queued it. Generate a correlation identifier at a trusted boundary, carry it with the job, and include it in structured worker logs. Keep the job identifier separate so that several jobs from one request remain distinguishable.

Carry context with a synthetic job

This standard-library example models the data passed to a worker. It does not run a broker or install distributed tracing.

from uuid import uuid4

request_id = uuid4().hex
job = {"job_id": "fixture-job-7", "request_id": request_id}
event = {"event": "job_failed", **job, "error_code": "source_missing"}
assert event["request_id"] == request_id
assert event["job_id"] == "fixture-job-7"

The identifier helps locate related events without copying the entire request into logs. If a client supplies its own identifier, validate its format and length before accepting or recording it. Otherwise a hostile value can create noisy logs or ambiguous records.

The request identifier travels with the queued job while each worker attempt keeps its own job details.

Keep retries visible

Use the same logical job identifier for a retry and record a separate attempt number. A request that queues several jobs needs several job identifiers under the same request context. When a workflow starts new independent work, decide whether that work belongs to the original trace or should link to it as a separate trace.

OpenTelemetry describes spans as units of work and span context as the data propagated across boundaries. A correlation field alone is useful, but it does not create span timing, parent relationships or automatic broker instrumentation. If the application uses a tracing SDK, use its documented propagation format instead of inventing a lookalike trace header.

Check the missing-link failure

In a local queue exercise, submit a known fixture request, force the worker to fail, and search for its request identifier. If the API log has the value but the worker does not, inspect the enqueue payload and consumer wrapper. If a retry loses it, inspect the retry construction separately.

Exclude authorization headers, customer message bodies and provider tokens from context fields. Log a controlled error category and a reference that an authorized operator can inspect elsewhere. Sampling and retention can also remove events, so an absent trace is not proof that a job never ran. Reference: OpenTelemetry traces. Related queue topics appear in the engineering blog.