northsun · ci · testing · reliability
Four more ways a check goes green without running.
Short answer: a check that cannot do its job still has to end somehow, and the easiest way to end is exit 0, which CI reads as green. Give "I could not look" a result of its own, different from "I looked and found nothing", and make the check count what exists before it filters anything. Then break its input on purpose and read the result against what you documented. Four checks in Northsun's repository could go green without running. One did, in use, according to its own header. The records describe the other mechanisms as found in review, or by their authors soon after, and fixed. Each is reproduced below in a few lines of Node, with the fix the records describe.
Status of what this describes: tooling and test code in Northsun's private repository, all of it on its main branch by 2 October 2026. The entry-point incident is dated August 2026 in its header. This post does not say where any of it runs. Nothing here is a claim about product behaviour, and nothing says which checks run on which hosts or that any host is protected. The script in this post is a stand-in with toy checks of its own, described below.
Who did the work: not me, as far as I can show. The merge and squash commits for the three pull requests carry my account name, and the commit messages behind them carry an AI co-author trailer. The commit that introduced the entry-point helper is outside the shallow checkout that was read for this post, so I make no claim about who wrote it. Of the three decision records cited here, two carry no provenance line and one names an AI coding agent as its decider. All three speak in the first person, so I report their sentences as theirs and do not claim them. The review findings the records credit came from automated reviewers. The script and the runs of it in this post were done on 7 October 2026 by the AI session that drafts these write-ups, not by me and not by whoever wrote the checks. Where this post says "Northsun", it means that project and its agents.
Terms, as this post uses them. An exit code is the number a program returns when it ends, and CI treats 0 as green. An entry-point guard is the test a script runs to decide whether it was started directly, so that importing it does not run its work. To fail closed is to treat a failure to check as a failure. A denominator check compares how many things a check classified with how many there were.
What happened
Northsun is the memory and enrichment layer for AI agents.
Northsun's repository is full of checks written as small scripts: gates that exit 0 for green, a sweep that looks for red workflows, evaluation gates with ceilings. The site has already told several stories about checks that pass without checking. A skipped job satisfied a required check, and a mutation was absorbed by operator precedence, in how 29 review rounds hardened a CI gate that nothing ran. Optional chaining on a missing provider made an analytics call that could not fail in the analytics call that couldn't fail. A test that counted the wrong thing and read zero for its own burst is in a load test that counted its neighbours. These four are different mechanisms, and the repository's records describe each one.
One: an entry-point guard that never matches
The header of the repository's entry-point helper dates the incident to August 2026. Scripts across the repository had hand-rolled the same guard, comparing import.meta.url with the path Node was started with. Node resolves a module entry point through symlinks, so import.meta.url holds the real path, while process.argv[1] keeps its symlinks and is made absolute. On macOS, /tmp is a symlink to /private/tmp, so any script run from a path under /tmp compared unequal. Its main function never ran, the module fell off the end, and the process exited 0. A phase gate that treats exit 0 as green reported clean without validating anything from a worktree under /tmp.
The fix, per the header, is to resolve both sides through realpath before comparing. The helper's own comment adds that it falls back to the unresolved path instead of throwing when a path cannot be read. The header also cautions that resolving only one side brings the bug back under Node's --preserve-symlinks-main flag, and section 1 of the script runs exactly that: under the flag, the guard that resolves only argv[1] fails and the one that compares as typed passes.
Section 1 reproduces the shape with a check that exits 1 when it finds a problem, run by its real path, through a symlinked directory, and through a symlinked directory with the flag.
Two: a gate that stops gating when its ceiling is missing
An evaluation gate asserted that a measured latency stayed under a ceiling read from a budgets file. The run could be pointed at another budgets file, and a legacy one carried keys for a different percentile. The key the gate looked for was then undefined, and measured > undefined is false for every measured value, so every latency gate would silently have stopped gating. The comment in the code puts it plainly: a gate that quietly stops gating is worse than no gate. The decision record says the problem was found in review of the pull request, and calls it a real fail-open that the pull request itself had introduced.
The fix makes a missing ceiling a breach, with the message that it is refusing to run ungated. Section 2 of the script shows a gate with and without the fix against a budgets object that lacks the key.
Three: a monitor that cannot tell blind from clean
The sweep is a script used to decide whether any gate is red. Review found that it could not tell total blindness from a clean sweep. It ran with set -uo pipefail and no errexit, so the script carried on after a failed API call that had still left an empty file, because the shell's redirection creates the file first. The script then counted zero things, classified zero things, and passed its denominator check because zero equals zero. Nothing was red, so it printed (no RED, no UNKNOWN) and exited 0. Per the commit message, an earlier version existed only on one machine until the script was committed, and is described as having reported (no RED, no UNKNOWN) and exit 0 while completely blind. The records do not say whether it ever did so outside review. The record demonstrates the failure by pointing the script's API client at /usr/bin/false. Before the fix: that line and exit 0. After: a verdict that the sweep could not read what it classifies, and exit 2.
Fixing that exposed a second problem, in the exit codes. A failed per-workflow read had first been counted as unknown and reported with exit 1, the code that means "I looked at everything and found red", so a caller could not tell a genuinely red gate from a sweep that could not see. The record says the API call's status was being discarded in command substitution, so a separate counter now carries read failures. In bash, local x=$(cmd) is the familiar way to lose that status: the line returns the status of local, which succeeds, and a plain x=$(cmd) keeps the status of cmd. A pipeline loses a failing stage's status only when pipefail is off, and the sweep ran with it set, so a plain pipeline would not explain the lost status. The script also checks the status of each read and not only the shape of its output, because the API client can print one valid row and then fail, and a check on shape cannot see that. The documented contract is exit 0 for all clear, 1 for "I read everything and found RED or UNKNOWN", and 2 for "the sweep could not be trusted", which is never clean.
The record also admits how its own demonstration was first misread: the fail-closed demonstration printed exit=1 and was read as a pass, two screens below a header documenting exit 2 for exactly that case. In its words, having the measurement is not the same as checking it against the spec.
Section 3 of the script has three sweeps over a world where one of three gates is red: one that swallows read failures, one that notices them but reports them with the same code as red, and one that gives them their own. The toy has no denominator check, so the 0 == 0 pass is reported from the record and not shown. Three cells of its table, the per-gate failure under the blind sweep, the unreadable list under the one-code sweep, and the empty list, are the toy's own extension.
Four: a denominator that shrinks with the numerator, and an absence read as intent
The same sweep filtered out workflows that were not active before it computed its total. The denominator check then compared two numbers that had been reduced in the same way, and it would have passed while a gate sat disabled and unable to run. The record calls that latent, not hypothetical. Review found it, and the fix is to enumerate every workflow first. A disabled workflow is never green, and a deleted one is counted and reported instead of dropped.
A later change fixed a cousin of the same mistake. The sweep had classified a workflow with zero runs as not applicable, and its header described that state as "PR-only by design". The change's commit message says the script cannot see design: it observed an absence of runs and asserted an intent from it. It added an ORPHANED state for a workflow that is active, whose definition path is under the workflows directory but absent from the branch's own listing. The reference set is one read of that listing, with membership tested locally instead of one request per workflow. Per the record, trust in that read is decided by its status and never by whether it is empty. An HTTP 200 is accepted as the truth even when the listing is empty. So is a confirmed not-found when the branch name still resolves, because a branch with no workflows directory is a valid state and failing closed on it would block exactly the case the new state exists to detect. A not-found where the branch name does not resolve, or any other failed read, ends the sweep as untrusted, because without the listing every workflow would look orphaned. The record says its first version failed closed on a read error and on an empty listing, and that both halves were wrong. Separately, an enumeration that comes back with zero workflows is still treated as a failed read and not as an empty repository.
Section 4 of the script counts four gates, one of them disabled, once by filtering before the count and once by putting every gate into a bucket first. The second also treats a disabled gate as never green, so two changes are in it. It then reads a zero-run gate as intent and as absence.
The script
It uses only Node built-ins, and it was run on Node 22.22.0 and 22.22.2 on Linux. It makes a scratch directory with a symlink for section 1 and removes it when section 1 finishes.
// repro.mjs: four ways a check goes green without running. Node built-ins only, no dependencies.
// Run: node repro.mjs
import { spawnSync } from "node:child_process";
import { mkdtempSync, mkdirSync, writeFileSync, symlinkSync, realpathSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
// ---- 1. An entry-point guard that never matches --------------------------------------------------------------
// A check script that only does its work when it is the entry point. The work "finds a problem", so it exits 1.
const GUARDS = {
"compares as typed": `import { pathToFileURL } from "node:url";
const isMain = import.meta.url === pathToFileURL(process.argv[1]).href;`,
"resolves argv only": `import { pathToFileURL } from "node:url"; import { realpathSync } from "node:fs";
const isMain = import.meta.url === pathToFileURL(realpathSync(process.argv[1])).href;`,
"resolves both sides": `import { fileURLToPath } from "node:url"; import { realpathSync } from "node:fs";
const isMain = realpathSync(fileURLToPath(import.meta.url)) === realpathSync(process.argv[1]);`,
};
const root = realpathSync(mkdtempSync(join(tmpdir(), "guard-")));
mkdirSync(join(root, "real"));
symlinkSync(join(root, "real"), join(root, "alias"), "dir"); // like /tmp being a symlink to /private/tmp on macOS
const run = (file, flags = []) => {
const r = spawnSync(process.execPath, [...flags, file], { encoding: "utf8" });
return `exit ${r.status} ${r.stdout.includes("check ran") ? "(ran)" : "(did not run)"}`;
};
console.log("1. entry-point guard; the check exits 1 when it finds a problem, so exit 0 reads as green:");
console.log(" " + "guard".padEnd(22) + "real path".padEnd(18) + "symlinked dir".padEnd(26) + "symlinked dir with --preserve-symlinks-main");
try {
for (const [name, guard] of Object.entries(GUARDS)) {
writeFileSync(join(root, "real", "check.mjs"), `${guard}\nif (isMain) { console.log("check ran: problem found"); process.exit(1); }\n`);
const real = join(root, "real", "check.mjs"), alias = join(root, "alias", "check.mjs");
console.log(" " + name.padEnd(22) + run(real).padEnd(18) + run(alias).padEnd(26) + run(alias, ["--preserve-symlinks-main"]));
}
} finally { rmSync(root, { recursive: true, force: true }); } // the scratch directory this script made
// ---- 2. A gate that stops gating when its ceiling is missing -------------------------------------------------
console.log("\n2. latency gate, measured value 900, budgets that may lack the key:");
const gateNaive = (measured, budgets) => (measured > budgets.limit_ms ? "BREACH" : "ok");
const gateClosed = (measured, budgets) => (typeof budgets.limit_ms !== "number" ? "BREACH (ceiling missing, refusing to run ungated)" : measured > budgets.limit_ms ? "BREACH" : "ok");
for (const [label, budgets] of [["budgets with limit_ms: 300", { limit_ms: 300 }], ["legacy budgets, other key only", { old_limit_ms: 300 }]])
console.log(` ${label.padEnd(34)} naive: ${gateNaive(900, budgets).padEnd(7)} fail closed: ${gateClosed(900, budgets)}`);
// ---- 3. A monitor that cannot tell blind from clean ----------------------------------------------------------
// A sweep reads a list of gates, then reads each gate's last result. Exit 0 = all clear, 1 = something is red, 2 = could not look.
// This toy has no denominator check: the record's 0 == 0 pass is reported in the post, not shown here.
const world = (broken) => ({
list: () => { if (broken === "list") throw new Error("api down"); if (broken === "empty") return []; return ["build", "deploy", "nightly"]; },
last: (g) => { if (broken === "reads") throw new Error("api down"); return g === "deploy" ? "failure" : "success"; },
});
function sweepBlind(w) { // swallows read failures: an empty read looks like an empty world
let gates = []; try { gates = w.list(); } catch {}
const results = gates.map((g) => { try { return w.last(g); } catch { return undefined; } }).filter(Boolean);
const red = results.filter((r) => r === "failure");
return { exit: red.length ? 1 : 0, text: red.length ? "RED" : "(no RED)" };
}
function sweepOneCode(w) { // notices failed reads but reports them with the same code as red
let gates; try { gates = w.list(); } catch { return { exit: 1, text: "could not read the list" }; }
let bad = 0; for (const g of gates) { try { if (w.last(g) === "failure") bad++; } catch { bad++; } }
return { exit: bad ? 1 : 0, text: bad ? "not green" : "all clear" };
}
function sweepHonest(w) {
let gates; try { gates = w.list(); } catch { return { exit: 2, text: "UNTRUSTED" }; }
if (!gates.length) return { exit: 2, text: "UNTRUSTED" }; // zero gates enumerated is a failed read, not an empty repo
let readFailures = 0, red = 0;
for (const g of gates) { try { if (w.last(g) === "failure") red++; } catch { readFailures++; } }
if (readFailures) return { exit: 2, text: "UNTRUSTED" };
return { exit: red ? 1 : 0, text: red ? "RED" : "all clear" };
}
console.log("\n3. sweep over three gates, one of them red; the API is broken in different ways:");
console.log(" " + "situation".padEnd(27) + "blind".padEnd(19) + "one code for both".padEnd(32) + "honest");
for (const [label, broken] of [["API healthy", null], ["list unreadable", "list"], ["list comes back empty", "empty"], ["per-gate reads unreadable", "reads"]]) {
const cell = (r) => `exit ${r.exit} ${r.text}`;
console.log(" " + label.padEnd(27) + cell(sweepBlind(world(broken))).padEnd(19) + cell(sweepOneCode(world(broken))).padEnd(32) + cell(sweepHonest(world(broken))));
}
// ---- 4. A denominator that shrinks with the numerator --------------------------------------------------------
console.log("\n4. four gates, one of them disabled so it can never run:");
const gates = [{ n: "build", state: "active", last: "success" }, { n: "deploy", state: "active", last: "success" },
{ n: "nightly", state: "disabled", last: undefined }, { n: "lint", state: "active", last: "success" }];
const bucket = (g) => (g.state !== "active" ? "disabled" : g.last === "success" ? "green" : g.last ? "red" : "unknown");
const denominatorNaive = (all) => { const active = all.filter((g) => g.state === "active"); // filtered BEFORE counting
const total = active.length, classified = active.filter((g) => g.last).length;
return { exit: total === classified ? 0 : 2, text: `total=${total} classified=${classified}` }; };
const denominatorHonest = (all) => { // every gate lands in exactly one bucket
const counts = {}; for (const g of all) counts[bucket(g)] = (counts[bucket(g)] ?? 0) + 1;
const classified = ["green", "red", "unknown", "disabled"].reduce((a, k) => a + (counts[k] ?? 0), 0), notGreen = all.filter((g) => bucket(g) !== "green");
if (classified !== all.length) return { exit: 2, text: "UNTRUSTED: a gate fell out of the count" };
return { exit: notGreen.length ? 1 : 0, text: `total=${all.length} classified=${classified} not green: ${notGreen.map((g) => `${g.n} (${bucket(g)})`).join(", ") || "none"}` }; };
for (const [name, f] of [["filter, then count", denominatorNaive], ["count, then bucket (disabled is never green)", denominatorHonest]]) { const r = f(gates); console.log(` ${name.padEnd(45)} exit ${r.exit} ${r.text}`); }
console.log(" and a gate with zero runs, read as intent or as absence:");
const defined = new Set(["build", "deploy", "lint"]); // the definition files that exist on the branch
for (const g of [{ n: "lint", runs: 0 }, { n: "ghost", runs: 0 }]) {
const asIntent = "N/A (PR-only by design)", asAbsence = defined.has(g.n) ? "N/A (definition exists, no runs yet)" : "ORPHANED (active, but its definition is not on the branch)";
console.log(` ${g.n.padEnd(6)} naive: ${asIntent.padEnd(26)} checks the definition: ${asAbsence}`);
}
Output from Node 22.22.0 on Linux. The run on Node 22.22.2 printed the same text.
1. entry-point guard; the check exits 1 when it finds a problem, so exit 0 reads as green:
guard real path symlinked dir symlinked dir with --preserve-symlinks-main
compares as typed exit 1 (ran) exit 0 (did not run) exit 1 (ran)
resolves argv only exit 1 (ran) exit 1 (ran) exit 0 (did not run)
resolves both sides exit 1 (ran) exit 1 (ran) exit 1 (ran)
2. latency gate, measured value 900, budgets that may lack the key:
budgets with limit_ms: 300 naive: BREACH fail closed: BREACH
legacy budgets, other key only naive: ok fail closed: BREACH (ceiling missing, refusing to run ungated)
3. sweep over three gates, one of them red; the API is broken in different ways:
situation blind one code for both honest
API healthy exit 1 RED exit 1 not green exit 1 RED
list unreadable exit 0 (no RED) exit 1 could not read the list exit 2 UNTRUSTED
list comes back empty exit 0 (no RED) exit 0 all clear exit 2 UNTRUSTED
per-gate reads unreadable exit 0 (no RED) exit 1 not green exit 2 UNTRUSTED
4. four gates, one of them disabled so it can never run:
filter, then count exit 0 total=3 classified=3
count, then bucket (disabled is never green) exit 1 total=4 classified=4 not green: nightly (disabled)
and a gate with zero runs, read as intent or as absence:
lint naive: N/A (PR-only by design) checks the definition: N/A (definition exists, no runs yet)
ghost naive: N/A (PR-only by design) checks the definition: ORPHANED (active, but its definition is not on the branch)
What it shows, with its limits. Section 1: a guard that compares as typed exits 0 without running through a symlinked directory. A guard that resolves both sides runs in all three setups. A guard that resolves only argv[1] fails under --preserve-symlinks-main, which matches the caution in the header. This is a Linux run with a symlinked directory, and the incident in the record was a macOS /tmp. Section 2: a missing key makes an unguarded comparison read as passing, and the guard turns it into a breach. Section 3: only the sweep with its own exit code lets a caller tell a red gate from a sweep that could not look, and an empty list is the case where the one-code sweep reports all clear. Section 4: a count taken after the filter passes with a disabled gate in the world, and counting first with disabled never green does not, and a zero-run gate can only be told from an orphan if something checks that its definition exists. It does not show how Northsun's sweep, gate or guard behave today.
What the four have in common
Each check had a default that meant success, and each could not tell "nothing to report" from "could not see". The records' fixes share a few moves, and not every record says all of them. Make "could not look" a result of its own, with its own exit code, and never let it share a code with "found a problem". Fail closed on a missing input and on a failed read. Whether an empty answer is a failure depends on what you know: the sweep treats an enumeration of zero workflows as a failed read, and a successful but empty listing of workflow files as the truth. Count what exists before filtering, so the denominator does not shrink with the numerator. The sweep's records add two more: never infer a design from an absence, and demonstrate the fix by breaking the input on purpose.
What this still does not prove
- The script is not Northsun's code. Its toy checks stand in for the patterns, and its output says nothing about how any Northsun tool behaves today.
- The sweep's own tests are not in the repository this post read, so its behaviour is reported from its header, its commit messages and its decision records, and the pattern is reproduced in a toy and not by running the sweep.
- The repro ran on Linux under two Node versions. The entry-point incident happened on macOS, and no macOS run is shown.
- All of it is tooling and test code on the main branch. The post says nothing about where any of it runs, and makes no claim that any host is protected.
- The review findings and the demonstrations in the records are as the records report them. They were not re-run for this post, and the repository is private.
How to check your own checks
Five questions, in order. What does the check exit when it cannot read its input, and is that the same code as when it found nothing? Is the denominator computed from what exists or from what survived a filter? Does "unknown" have a result of its own, separate from red and from clean? Have you broken the check's input on purpose, with its API client pointed at a command that always fails, with a config key missing, and, if it compares paths, through a differently spelled path, and read each result against what you documented? And is any "by design" label inferred from an absence?
FAQ
Why exit 2 and not exit 1 for "could not look"?
Because exit 1 asserts a complete look. The sweep's header says it is never returned when a read failed, so a caller can tell "I read everything and found RED or UNKNOWN" from "the sweep could not be trusted".
Why not just turn on errexit?
The records name the missing errexit as part of the failure but do not weigh turning it on, so this is shell behaviour and not their finding. Errexit stops at the first failed read with that command's own status, which is not the code 2 the sweep reserves. It is ignored in if conditions, and local x=$(…) masks the status. For per-workflow reads the sweep keeps going, counts every read failure and returns 2, and an unreadable enumeration ends it at once with 2. The records add that checking the shape of the output was not enough: the API client can print one valid row and then fail, so the status of each read has to be checked too.
Is the symlink problem only a macOS problem?
The incident in the record was on macOS, where /tmp is a symlink. The script's section 1 reproduces the same failure on Linux with a symlinked directory, so any setup where scripts are started through a symlinked path can hit it.
How do I test that a check can fail?
Break its input on purpose and read the result against what the check documents: point its client at a command that always fails, remove a config key, run it through a symlink. The record's own demonstration did the first of these, and the record's own lesson is that printing the result is not the same as comparing it with the specification.
The receipts
The Northsun repository and issue tracker are private, so these are paths and merge commits for anyone with access to check. The decision records carry internal figures, which this post leaves out.
For the first stories of this kind, read how 29 review rounds hardened a CI gate that nothing ran, the analytics call that couldn't fail and a load test that counted its neighbours and never saw its own burst. A related one: a flaky timing test that blocked a deploy. A later one, about an empty array that meant four different things, is a model parser that returned the same empty array for four different things.