Our memory read kept missing its time budget. The last cause we found was one line of config.

Short answer: if a database read is fast inside the database but misses its time budget from a Cloudflare Worker, work out where the time goes before you buy more compute. Check the database side first: is the working set in memory, and does the plan show cache hits? If it is cheap, look at the network path. A Worker with no placement hint runs near whoever called it, so every round trip to a single-region database crosses that distance. Cloudflare's own docs give per-query figures for a distant region and a nearby one, and the gap is many times over (vendor-stated; read the figures there). Pin the Worker to the database's region with a placement hint, and test the result against predictions you wrote down before the deploy.

That one line was the last fix we made, not the first. Before it came a bigger database, and a lesson about what a timeout can and cannot tell you.

Status of what this describes: the change is merged and was deployed (checked against the live version endpoint when it shipped). The measurements are a small sample of our own dogfood traffic. The story starts from calls by one test caller, taken on the smaller database tier, so treat it as an anecdote and not a rate. This is an engineering account, not a benchmark and not a product claim.

Who did the work: Northsun's delivery agents did, not me. They are AI sessions (Claude for building and coordinating, with other models and review bots checking the changes) working under written rules: one owner per change, predictions before measurement, independent review. I moved the database to a bigger tier. I did not run the measurements, and they were not re-run for this post. Where this post says "we", it means the agents and the Northsun project. The record shows the agents choosing the explicit placement hint and deciding not to revert it; those were their calls.

What happened

Northsun is the memory and enrichment layer for AI agents.

On our own deployment, where the governed read is switched on, its voice route answers a call-start request with a returning caller's stored facts. Voice is the hard case for memory, because a person is waiting on the line. So the read of those facts runs under a hard time budget, and the route fails open: if the read has not finished when the budget runs out, the response goes out with facts: null and the call carries on without the memory. Northsun is designed for sub-300ms voice recall. That is a design target, not a measured claim, and this post is about what the first measurements taught us.

A synthetic probe had reported an earlier, two-statement form of the read as incomplete. The read was then folded into one statement, which is one round trip and the right shape. On the build that carried it, the probe still reported incomplete. Calls from one returning test caller on our dogfood tenant also came back without facts. That sample is anecdotal: its own counts were later flagged because they did not reconcile, so it establishes no rate. What stood out was how the reads behaved: they did not fail. They finished, late.

What the code told us, and what it could not

Failing open was not silent. A read that lost the race was reported as a failure: a warning in the logs, an event in the error tracker, and a red probe. Returning null instead of an empty list was a deliberate signal that the read had not completed. What nobody could see was how slow the lost read had been. The timer won the race, and the read's result and its elapsed time were thrown away.

That mattered because our rule is that the budget figure changes only on a production measurement, and for a read that always lost the race, that measurement could not exist. So the first addition was an instrument: record the real elapsed time of the read on every settlement, including one that lands after the budget has fired. In our case it shipped in the same change that collapsed the read into one statement. Here is the idea, simplified and not our production code:

type Log = (line: Record<string, unknown>) => void;

export async function readWithBudget<T>(
  read: () => Promise<T>,
  budgetMs: number,
  log: Log,
  hold: (p: Promise<unknown>) => void = () => {}, // optional observer of the read's promise
): Promise<T | null> {
  const startedAt = Date.now();
  let budgetFired = false;
  const line = (result: string) =>
    log({
      stage: "read.timing",
      result,
      elapsed_ms: Date.now() - startedAt,
      budget_ms: budgetMs,
      budget_fired: budgetFired,
    });

  let timer: ReturnType<typeof setTimeout> | undefined;
  const timeout = new Promise<null>((resolve) => {
    timer = setTimeout(() => {
      budgetFired = true; // set by the timer itself, not re-derived from clocks
      line("budget_fired"); // a read that never settles still leaves a record
      resolve(null); // fail open: the caller carries on without the data
    }, budgetMs);
  });

  // Observe EVERY settlement, including one that lands after the budget fired.
  const observed = Promise.resolve()
    .then(read)
    .then(
      (value) => {
        line("settled");
        return value;
      },
      (error) => {
        line("errored");
        throw error;
      },
    );
  hold(observed.catch(() => {})); // tracks the read until it settles

  try {
    return await Promise.race([observed, timeout]);
  } catch {
    return null; // a failing read must not take the response down with it
  } finally {
    clearTimeout(timer);
  }
}

It was run under Node 22 against five stand-in reads: one that finishes well inside the budget, one that finishes well after it, one that fails after it, one that never finishes, and one that throws at once:

Two differences from our production code. It logs timing only when a read settles, so the line at the timer is an addition here. And our production code never holds the read itself, only the error report; in this listing, hold tracks the read until it settles. Do not pass that promise to an invocation-lifetime hook such as ctx.waitUntil unless the read is bounded: a read that never settles can keep the invocation alive until the platform limit.

We suspected the database first

With every read reporting its time, we looked at the database side first. The working theory was that on the smaller database tier the hot working set did not fit in the database's memory cache, so the read did physical I/O. I moved the database up a tier. We never captured a query plan from before the upgrade, so we cannot say how much of the early misses that explained. After the upgrade, the read's plan for our hottest caller showed an index scan with every page a cache hit, and an execution time that was a small fraction of the budget. That is recorded in the investigation and was not re-run for this post. So the tier was a contributing cause of unknown size, and the database side was now cheap.

The budget kept firing for some requests. In one capture window after the upgrade, against the same database and in the same hour, requests that entered near the database were within budget, and requests that entered at another location were not. The scheduled probe, which does not enter through the edge at all, still missed too.

The second suspect: the Worker's distance from its database

That contrast pointed at the path between the Worker and the database. The Worker had no placement hint, so Cloudflare ran it close to whoever called. The database sat in one region. Same database, same hour, a different distance.

before   caller ─► edge near caller ─► Worker runs here ─► auth ─► contact ─► transaction ─► [budgeted read]
                                                           every step is a round trip to Postgres, in one region

after    caller ─► edge near caller ─► Worker runs beside the database ─► the same steps, short hops

The budgeted read is one statement, so one round trip. The route as a whole makes several sequential round trips: authenticate the key, look up the contact, set up the transaction, then the read. The budget covers only the last one. We did not break the effect down per hop. Our own estimate was that the far entry paid about two extra cross-region round trips, and that is an inference. It sits uneasily with the read being one statement, so the exact mechanism is unexplained. What we can say is that the near and far contrast pointed at distance, and that placing the Worker beside the database fixed it, per the before and after result below, and nothing more.

Cloudflare's guidance has a limit worth knowing: "If your Worker makes only one query per request, placement does not improve end-to-end latency." Our budgeted read is one query. Our route makes several.

The fix, in wrangler.jsonc:

{
  // Run the Worker in the same cloud region as the database behind the pooler.
  // The value is a placeholder: use the provider and region your database reports.
  "placement": { "region": "<provider>:<database-region>" }
}

We chose an explicit hint over Smart Placement. Cloudflare describes Smart Placement as needing consistent traffic from multiple locations, with analysis taking up to 15 minutes. For a service with modest traffic it might never activate. A region hint is deterministic. The change is reversible: remove the key and redeploy. We also rejected two other moves. One was another database tier, because the database read was already cheap when warm. The other was raising the budget first, because a budget locked in on a misplaced Worker describes a misconfiguration.

Testing it: predictions, a witness and a control

We wrote the predictions down before the deploy, so the measurement could refute them: (a) after the deploy, every request-path row should look like the rows that already entered nearest the database; (b) if that holds, the budget needs no amendment, and the new environment variable stays as a safety valve; (c) an open question, whether a call that a scheduled handler makes through a service binding inherits placement.

The evidence came from three populations, kept apart:

The method was read-only: tail the Worker's logs as JSON, filter on the timing line, and pull the entry location, elapsed_ms and budget_fired from each row.

wrangler tail <worker-name> --format json --search "read.timing"

What we observed. In a 90-minute window after the deploy, the budget did not fire on any request-path row we captured. That included the entry location whose requests had been missing the budget before. The rows that entered nearest the database, which should not change, did not. The scheduled probe also stopped missing the budget, and its first runs after the deploy served the facts where the runs before had not. That means it was not an independent control after all. Our reading is that the call it makes through a service binding inherited placement, an inference from a timing change on a few rows. We did not check for a placement header, so this is inference. The budget figure stayed where it was.

A control that moves is not a control. Check that yours is untreated before you lean on it.

What this does not show. The result supports the hypothesis. It does not prove it. It is a small sample at the entry location that mattered, on dogfood traffic, from different hours of the night and different builds. The window spanned three builds, and another change to the same route landed in the middle, although the read cleared the budget on both sides of it. The timing is the Worker's read time, not what a caller experiences. We are not publishing our own timings: they are internal measurements from a small sample, and what we can stand behind is the method.

The cost on the other side

Pinning a Worker beside its database moves the distance to a different leg. The Worker-to-database hop gets short, and the caller-to-Worker hop gets longer for anyone far from that region. That is the trade, and it is true of any pinned Worker.

We first wrote the acceptance up as met. Two automated review passes flagged that the issue's own acceptance named caller-perceived latency, while a Worker-side tail measures only the read. The verdict was corrected to partial, and the open item stayed open until we measured the far leg. We did that with a public probe network: plain HTTP requests to a public health endpoint from several regions, with no credentials and no customer data.

We had also written a rule before measuring: if far callers crossed the caller-latency line we had named in advance, we would revert the pin to something traffic-adaptive. The rule fired: the probe's far-region results crossed that line. The rule named a total for the recall route, and the probe could only fetch a public health endpoint, so it was a proxy.

We did not revert, and the acceptance item was closed. The argument was that all the traffic we had captured was our own testing, and none of it came from the far regions we probed, so no far callers exist yet. That is a judgment about a population we have not observed, not a result. The record shows the agents making that call. We also found that the remedies we had written down in advance, traffic-adaptive placement or a second Worker, would not be expected to help while the database stays in one region. So we wrote the constraint down instead. For callers far from the pinned region, our own probe put the proxy measurement over the line we had set, so the design target above is not something we can claim for them.

The consequence worth taking away: a second Worker in another region would not be expected to fix far callers while the database and its pooler stay where they are, because the far Worker would just recreate the cross-region read. To serve far callers, expect to move the data, not just the Worker. That is an engineering consequence we recorded, not a product commitment.

How to check your own Worker

1. Time every settlement, not just the winner. If you fail open, log the elapsed time of the read that lost the race. A lost race without a time tells you something failed, not by how much.
2. Check the database side first. Does the working set fit in memory? Does EXPLAIN (ANALYZE, BUFFERS) show cache hits and no disk reads? If not, fix that before the network.
3. Find where the database lives and where the Worker runs. Cloudflare's docs describe a cf-placement header for checking placement, and warn it may be removed before Smart Placement leaves beta.
4. Count the sequential queries on the hot path. With one query, placement may not improve end-to-end caller latency, but it can still help a budget on that one read by shortening the Worker-to-database round trip. With several queries, it can matter a lot end to end.
5. Choose the hint to match the topology. One backend in a known cloud region and modest traffic: an explicit region hint. Several backends and steady traffic from many places: Smart Placement.
6. Write the predictions before you deploy. Include the result that would make you revert.
7. Keep fetch and scheduled rows apart, and check your control. A control that moves with the treatment is not a control.
8. Measure the far side too. A public probe network will tell you what callers in other regions pay, without touching your secrets.

FAQ

Why is my Cloudflare Worker slow to query Postgres when the query is fast in the database?

Often the query is not the slow part. Check the database side first: memory, cache hits, plan. Then look at the network. Each query is a round trip between where the Worker runs and where the database lives, and Cloudflare's guidance gives per-query figures for a distant region and for a Worker placed nearby, with a large gap. Several sequential queries multiply it. Measure the round trip before you scale the database further.

Does Smart Placement work for a low-traffic Worker?

It may not activate. Cloudflare says Smart Placement needs consistent traffic from multiple locations and up to 15 minutes of analysis. For a single backend in a known cloud region, an explicit region hint avoids the wait and the traffic requirement.

Does placement apply to cron (scheduled) handlers?

Cloudflare says placement affects only fetch handlers, and does not affect RPC methods or named entrypoints. We treated the scheduled invocation as unplaced. In our deployment, the synthetic call a scheduled handler makes through a service binding appeared to be placed, judging by a timing change on a few rows. We did not look for a placement header, so that is an inference on our setup. Test yours.

Should I raise my timeout budget instead?

Not first. A budget is a statement about what the caller can wait for. Raising it to fit a misplaced Worker hides the cost without paying it. Find what the budget is actually measuring, fix that, and then decide whether the figure needs to change.

Is a fail-open timeout a bad pattern?

No. Failing open is the right call when a slow read must not hold up a person on the line. Ours was not silent, either: failures were logged and the probe went red. What it lacked was the elapsed time of the read that lost. Keep the behaviour and record every settlement.

The receipts

The Northsun repository and issue tracker are private, so these are paths and merge commits for anyone with access to check. The decision record carries the internal figures, which this post leaves out.

The deploy was checked against the live version endpoint at the time. The production measurements were not re-run for this post.

For the argument that came before these measurements, read the latency objection for voice-AI memory is a dead argument. For what a memory layer has to do before latency matters at all, read What is an agent memory layer?. If you want the next field report in your inbox, subscribe here.