Our search endpoint's p99 fell off a cliff on a Tuesday afternoon. The edge dashboard was calm: request rate per IP flat, 429s near zero, WAF quiet. Postgres was logging FATAL: remaining connection slots are reserved, and every service on that database was stuck waiting for a pool connection. I had spent the morning tuning the wrong layer.
Request rate and query cost are different units
The edge counts requests per IP per second. That is a scalar. One request equals one unit, no matter what the request does.
The database counts concurrent queries and what each one costs. A request that returns a memoized ID is nearly free. A request that scans a large table with LIKE '%term%', or runs a report with several joins and an unindexed ORDER BY, holds a backend process, memory, and often a lock for seconds. A thousand cheap requests and a dozen expensive ones are the same number to the edge and completely different numbers to Postgres. Your limit can be perfectly enforced and still irrelevant, because the thing you capped is not the thing that ran out.
The mismatch is worse than it looks. The edge limit is per second. The database ceiling is concurrent connections, a fixed integer set at boot, minus the superuser reserve. Once those slots are taken, no request gets one until another finishes. That is a queue with a hard wall, and the wall is reached by duration multiplied by concurrency, not by rate.
Per-IP limits do nothing against one valid token
A per-IP rule assumes the load is spread across many addresses. The incidents I have been paged for came from one client with one valid bearer token. A mobile app retrying a failed export. An internal job fanning a report out across every tenant. A customer's integration polling a list endpoint with a filter that defeats the index.
One IP. One identity. Well under any sane per-IP threshold. Each request burning hundreds of milliseconds of database time. The edge sees a polite client. The pool sees a stampede.
Caching is the real protection for expensive reads
If a query is expensive and its inputs repeat, do not throttle it. Do not run it. A cache protects the database because it removes the work instead of declining it.
The catch is the key. A cache keyed on the URL with a good global hit ratio can still miss on every request to the one endpoint that matters, because each request carries a different combination of sort, page, and date range. Key on the normalized query and put the filter set in the key. Bound the cardinality, or you have built a slower database with worse durability.
For reads that genuinely cannot be cached, the fallback is a cheaper query: an index that matches the filter, a materialized view refreshed on a schedule, a precomputed count. Measure before guessing. Run EXPLAIN (ANALYZE, BUFFERS) with real parameters and look at actual rows and actual time. Those two numbers decide whether the answer is a cache, an index, or an admission limit.
Put the limit where the resource is
The limit belongs next to the pool, not at the edge. A semaphore in front of the connection pool caps concurrency at the exact resource that runs out.
import asyncio
DB_CONCURRENCY = 32
db_gate = asyncio.Semaphore(DB_CONCURRENCY)
async def fetch_report(pool, query, params):
async with db_gate: # the real ceiling: concurrent queries
async with pool.acquire() as conn:
await conn.execute("SET LOCAL statement_timeout = '2s'")
return await conn.fetch(query, params)
Two things are doing the work here. The semaphore is a ceiling on concurrent queries, not on requests per second, so it fails closed exactly when the database would. statement_timeout means a query that goes wrong dies inside Postgres instead of holding a backend forever. Set it at the transaction level so it does not leak into the next checkout of that connection.
For anything heavier, add a cost-based admission check: ask the planner for an estimate, and if the projected cost exceeds a budget, push the job to a queue or reject it with a clear error rather than starting work you cannot finish.
Then instrument the resource, not the edge. Watch pg_stat_activity for state, wait event, and query start time. A long-running query is load. A session sitting in idle in transaction is a leak, and no rate limit will fix it.
The edge limit is still worth having
None of this makes the edge rule useless. It absorbs the dumb traffic: credential stuffing, scrapers, a retry loop that never authenticates. It is cheap, it is global, and it keeps noise away from your application. Keep it.
Just stop treating it as the last line of defence. The edge limit protects the edge. The semaphore, the statement timeout, and the cache protect the database. Count the thing that runs out.
I write about production failures in Postgres, queues, and distributed systems.
Top comments (0)