Bound concurrent requests with asyncio in Python

Launching one task per input is easy to write and easy to overwhelm. A concurrency limit controls how many operations are active at once. It should sit around the operation that uses the limited resource, with a timeout for each attempt.

Requests entering a fixed-width semaphore queue.

Measure the limit with a local stand-in

This complete Python example uses sleeping coroutines instead of external HTTP requests. It demonstrates scheduling behavior without contacting a provider.

import asyncio

async def main():
    limit = asyncio.Semaphore(3)
    active = peak = 0

    async def fetch(item):
        nonlocal active, peak
        async with limit:
            active += 1
            peak = max(peak, active)
            try:
                async with asyncio.timeout(0.5):
                    await asyncio.sleep(0.02)
                    return {"id": item, "status": "ok"}
            finally:
                active -= 1

    async with asyncio.TaskGroup() as group:
        tasks = [group.create_task(fetch(i)) for i in range(12)]

    assert peak <= 3
    assert active == 0
    results = [task.result() for task in tasks]
    assert len(results) == 12
    print("peak:", peak)

asyncio.run(main())

The increment occurs after acquiring the semaphore, and the decrement lives in a finally block. Errors and cancellation therefore release the accounting as the semaphore context exits.

Replace the sleep with a genuinely asynchronous client call when integrating HTTP. A synchronous networking call inside this coroutine can block the event loop. Configure the client's connection pool and network timeouts as well, since waiting for a semaphore is only one part of the operation.

Concurrency differs from request rate

Three active requests can still produce many requests per second when responses are fast. A provider's per-minute quota needs a rate policy in addition to this concurrency bound. Retries should pass through the same controls so a failure burst cannot bypass them.

The example creates a task for every item, which is acceptable for twelve fixtures. For millions of records, use a bounded work queue and a fixed number of worker coroutines. Otherwise pending task objects can consume memory even while the semaphore limits active network calls.

Decide whether one failure should cancel the whole batch. TaskGroup provides a useful shared-lifetime model; collecting independent successes and failures requires a different result policy.

Postzyo provides context for bounded external work. See Python's task and timeout documentation for the cancellation semantics used here.