A model parser returned the same empty array for four different things.

Short answer: when a function that reads a model's answer can return an empty list for several different reasons, the empty list stops being data and becomes a place for failures to hide. Keep the leniency, because a bad model turn should not fail the job, but make the return value say which situation it was, store that next to the event, and log the situations that mean "I could not read this". Northsun's record describes a parser that returned an empty array from four situations in its code and could not tell any of them from the model genuinely finding nothing. The fix named eight outcomes and split them on one question: was the answer understood? A short Node script below reproduces the collapse, the fix, and four defects that review found in the fix or in code it inherited.

Status of what this describes: code, tests and a runbook in Northsun's private repository, merged to its main branch on 15 August 2026. I found no record that the change was deployed or checked in production, so this post says merged and nothing more. I looked in the Northsun git history and its tracker, and I did not read its deploy runs or its list of deployed versions. Nothing here is a claim about product behaviour, about how often any model gives an unreadable answer, or about what any user's data looks like. The script in this post is a stand-in with toy parsers of its own, described below.

Who did the work: not me, as far as I can show. The merge commit carries my account name, and the four non-merge commits behind it are authored under my name, each with an AI co-author trailer and a session trailer in its message. I cannot tell from the records who reviewed the three rounds, so I report the commit messages' sentences as theirs and do not claim them. The private tracker notes are posted under my account name but speak as an AI session, and they describe a check by another AI system that refuted the accepted diagnosis. The script and the runs of it in this post were done on 7 October 2026 by the AI session that drafts these write-ups, not by me and not by whoever wrote the parser. Where this post says "Northsun", it means that project and its agents.

Terms, as this post uses them. An outcome is a named result of reading a model's answer, such as "the answer was cut off". Understood means the answer was read and said something, even if what it said was "nothing here". A receipt is a small record stored with an event that says how its answer was read. A lenient parser returns something usable when its input is bad, instead of raising an error.

What happened

Northsun is the memory and enrichment layer for AI agents.

In the code the record describes, one step sends a transcript to a model and expects a JSON object back, with a memories list of things worth keeping. Nobody wanted a bad model turn to fail the whole job, so the parser that read the answer was lenient: whatever came back, it returned a list, and an answer it could not use became an empty one.

On 4 August 2026, per the commit message, two events completed on the pipeline's one terminal success status with no memory objects counted, while cost telemetry, in the runbook's phrase, showed that the model call had run and been billed. Those event facts rest on the commit message and the tracker, and an AI-written peer-review note that landed in the repository the day after the merge (commit 262054d4) marks the event-level claim that they were billed and stored zero memory objects as not determinable from its session, for want of a production query. Whether that count included verbatim chunks is part of the puzzle under "What the fix did not do". That left a question the record could not answer. Either the model had read the transcript and found nothing worth keeping, which is correct, or it had produced facts and the parser had failed to read them, which is data loss. Nothing recorded the answer or the path the parser took, so whether facts were lost was, in the commit's words, "unfalsifiable after the fact". The runbook added with the fix says the same, and says it stays that way for any event processed before the fix, which would include those two.

The site has told stories of checks that pass without checking, in how 29 review rounds hardened a CI gate that nothing ran, the analytics call that couldn't fail and four more ways a check goes green without running. This is a cousin. What could not tell success from failure here was not a check but a return value.

Four routes to an empty array

The commit message says the old parser returned a bare [] from four different situations. Reading the old function, the parent of the fix commit, I count four routes to an empty return, and the script's first section has a stand-in with the same four. The parser looked for a JSON object in the reply, and if it found none it returned an empty list. If it found one and parsing threw, it returned an empty list, and so did anything else that threw while the answer was cleaned up, because the clean-up sat inside the same try block. If parsing worked but the memories key was missing, a default turned that into an empty list. And if the key held a list, the cleaned list was empty whenever the model had offered nothing or nothing usable.

Behind those four routes sit many more situations: an empty reply, a reply of prose, an answer cut off before its object closed, malformed JSON, valid JSON of the wrong shape, a list of unusable entries, and a model that genuinely answered "nothing here". The caller got the same [] from all of them. In section 1 of the script, eight of nine answers come back that way.

Eight outcomes, split on one question

The fix made the parser return an outcome and an understood flag instead of a bare list. The old entry point still exists as a thin wrapper that returns only the candidate list, for callers that never wanted more, according to its own comment; the extraction step calls the new one, so only callers of the new entry point see the vocabulary. Three outcomes are understood. ok means at least one usable entry. empty_result means the answer said memories: [], so the model answered "nothing here". all_candidates_invalid means the model offered entries and the clean-up rejected every one. Five are not understood, and the code comments say loss is possible there: empty_response, no_json_found for prose, unbalanced_json for an object that opened and never closed, malformed_json, and missing_memories_key for valid JSON of the wrong shape. The code comments call the split the one that matters to an operator: "we read the model's answer; the emptiness is a real answer", against "we could not read the answer; facts may have been lost here".

Two naming choices are worth copying. unbalanced_json names a shape and not a cause. The code comment calls truncation at the output cap the most common source, and I found no measurement behind that. A refusal that quotes the schema produces the same shape, and the commit for review round three records that the runbook had been telling operators to raise the cap for it. Naming a cause would have sent triage at a guess, which is the failure the vocabulary exists to stop. Other code comments still name truncation as the cause, one in the types file and two in the parser, so the repository is not fully consistent about it. And a missing answer is kept distinct from an empty one: the commit says that "we never called the provider" is not "the model said nothing".

The outcome also reaches storage. The merged code writes a receipt on each event's own row, whether or not the answer could be read, with the outcome, the understood flag, the length of the answer, and counts: entries the model offered, entries that survived clean-up, entries that survived scoring, and structured memories stored, with verbatim chunks counted separately. Some of those counts had been computed and then thrown away. For an unreadable answer it also logs a warning tagged with a counter name that an alert can key on, and there had been no log line at all before, which the commit gives as the reason the error tracker stayed clean. The commit says an operator can then answer "did this event lose facts?" from a stored row with no replay, and the runbook limits that to events processed by code that includes the change. For an answer that could not be read, the receipt can say only that loss was possible, because it holds no model text. With no deploy record, I cannot say that any event has been processed that way.

What stayed the same

The leniency. A bad model turn still never fails the job, and the runbook says this was decided and not defaulted. Among its reasons: it treats extraction at temperature zero on an identical prompt as repeatable, so an automatic retry repeats essentially the same computation and is billed again, and a retry that discards the output again is not a repair. The repair, in its words, is making the outcome visible so an operator can decide.

The privacy boundary, as the record states it. The receipt carries shapes, lengths, counts and outcome classes, and no model text. The parser discards the error object from a failed JSON.parse unread, because V8 can embed a snippet of the offending input in the message, and the runbook says a test pins that. Reading the tests, they assert that a receipt rebuilt from the parse result (the test calls it the stored receipt) and the logged warning do not contain the input, but the input in the test named for the parse-error message has an unclosed brace and never reaches JSON.parse, so I would not call the discard itself pinned. The runbook's heading says not to improve this by storing the response, and its way to get an answer back is to replay the stored transcript. Section 4 of the script prints two parse errors from the same engine: one holds a slice of the input and the other only a position. This post reports the boundary as the record describes it, and does not call it a guarantee.

What review caught in the fix

The merge note says the pull request merged after three review rounds. The first catch, described in the first fix commit (which does not say which round or reviewer found it), was a regression of the never-throws rule. Rounds two and three each found the change's own defect class again, one level down: outcomes that could not be told apart.

In the change itself. Splitting the outcomes moved the clean-up out of the try block that used to absorb its failures. A well-formed answer such as {"memories":[null]} would then have thrown and failed the job, which is exactly what the leniency forbids. The commit says review caught it, and an explicit filter on non-object entries restores the rule, with its own test. Section 4 shows the throw and the filter.

Round two, two things the commit calls outcomes that could not be told apart. A JSON-column update replaced a nested object where it should have merged, because the SQL operator merges only the top level. With the receipt now written on every delivery, a second attempt would have erased the first attempt's count, which the commit puts as the change destroying the evidence it exists to create. Its test runs against a real database, and the commit gives the reason: the defect is the operator's semantics, and a mock would assert a belief about the operator and not its behaviour. Separately, a count taken after scoring read zero both when the scorer rejected everything and when the clean-up did, and the runbook read zero as the clean-up having removed them all. A count before scoring separates the two. The commit says both tests were mutation-verified. Section 5 of the script shows the shape of both with toy code; the first uses an object spread, which has the same top-level behaviour, and does not run the SQL.

Round three, a shortcut that decided truncation by punctuation. A fast path accepted any reply that started with an opening brace and ended with a closing one, which tests the first and last characters and not whether the braces balance. A reply cut off just after a nested object closed satisfied both, reached the parser, failed, and was labelled malformed_json. The commit calls that truncation wearing the label whose operator fix is different, and says the input was measured returning malformed_json before the fix. The fast path is gone. The shortcut was inherited from the old parser, where it changed only which step failed and the result was still an empty list. It became a mislabel once the new vocabulary separated unbalanced from malformed, and the first two commits of the fix kept it. Section 3 reproduces it.

Round three also corrected the runbook, which carried four claims the work could not support. A threshold on output tokens was removed because nothing had measured it. A scope line saying "zero memory objects" was too strong, because the receipt's stored count leaves out verbatim chunks. A triage query cast a caller-influenced value to an integer, so one stray legacy value would have aborted the whole query. And the no-replay claim holds only for rows that carry a receipt. The corrections are in the commit message, and they are the part of this story that is easiest to skip and most worth copying.

What the fix did not do

The status name still reads as success. The commit and the runbook say the terminal status is the same for every outcome, including the unreadable ones, because widening it needs a database constraint migration, which the commit filed separately. The runbook's own words: "Until that lands, the status is not a reliable success signal; the receipt is." When I looked on 7 October the follow-up was still in the backlog, and the main branch's storage code still sets that one status when extraction finishes.

It also cannot answer the original question for the two events. The private tracker notes add a puzzle they leave open. A separate path stores the transcript as verbatim chunks whatever the model said, so an event with a non-empty transcript should not have ended with nothing stored. At least one of four things had to be false (the notes say exactly one, but the four are not mutually exclusive): the count had left out verbatim rows, the transcript was empty, the worker predated that path, or the rows had been removed. The notes say they could not determine which, and the model's raw answer had never been kept anywhere. The notes also say the earlier diagnosis, that the model returned nothing because the input held no people and persistence never ran, was not supported by the code, because the store step runs unconditionally after extraction. They keep the no-people idea only as a plausible contributing factor and not a mechanism. The comment that says this is headed "fix shipped" and was posted about three hours before the merge, so "shipped" there evidently means the pull request had been opened, not merged or deployed, and the issue was still In Review on 7 October. I report this from the notes and did not re-run any of it.

The script

It uses only Node built-ins, and it was run on Node 22.22.0 and 22.22.2 on Linux. It has two toy parsers with the same shape as the record's old and new ones, and nine model answers. It does not model replies wrapped in a code fence, which the real parser handles. Its scan also differs from the record's in one detail: it treats a backslash as an escape only inside a string, while the record's scan skips the character after a backslash wherever it appears. A separate AI session that fact-checked this post on 7 October 2026 ran copies of the old parser (at commit 77ee27c2) and the merged one (at 40d46a65), with their imports stubbed, on the same nine answers: the old one returned an empty list for the same eight, and the new one gave the stand-in's outcome and understood flag for all nine.

// repro.mjs: one empty array, four meanings.
// A stand-in for a step that asks a model for a JSON answer shaped {"memories":[...]} and reads it.
// It is not Northsun code. Run: node repro.mjs   (Node built-ins only)

// The answers a model can give. Only the last one holds a usable fact.
const ANSWERS = [
  ["found nothing",           '{"memories":[]}'],
  ["empty reply",             ""],
  ["prose, no object",        "There is nothing worth keeping here."],
  ["cut off in the object",   '{"memories":[{"kind":"fact","text":"likes tea"},{"kind":"fact","te'],
  ["malformed JSON",          '{"memories":[{"kind":"fact","text":"likes tea",}]}'],
  ["valid JSON, wrong shape", '{"facts":[{"text":"likes tea"}]}'],
  ["only blank entries",      '{"memories":[{"kind":"fact","text":"   "}]}'],
  ["a null entry",            '{"memories":[null]}'],
  ["one usable fact",         '{"memories":[{"kind":"fact","text":"likes tea"}]}'],
];
const usable = (c) => typeof c.text === "string" && c.text.trim().length > 0;
const pad = (s, n) => String(s).padEnd(n);

// Find the object inside the reply with a depth scan that respects strings.
// fastPath is a shortcut the old parser had, and the first two commits of the fix kept it: accept the reply if it starts with { and ends with }.
function locate(text, { fastPath }) {
  const t = text.trim();
  if (!t) return { kind: "absent" };
  if (fastPath && t.startsWith("{") && t.endsWith("}")) return { kind: "found", text: t };
  const start = t.indexOf("{");
  if (start === -1) return { kind: "absent" };
  let depth = 0, inString = false, escaped = false;
  for (let i = start; i < t.length; i++) {
    const ch = t[i];
    if (escaped) { escaped = false; continue; }
    if (inString && ch === "\\") { escaped = true; continue; }   // an escape only means something inside a string
    if (ch === '"') { inString = !inString; continue; }
    if (inString) continue;
    if (ch === "{") depth++;
    if (ch === "}" && --depth === 0) return { kind: "found", text: t.slice(start, i + 1) };
  }
  return { kind: "unbalanced" };
}

// Version A, the old shape: always an array, and every failure is an empty one.
function parseOld(text) {
  const found = locate(text, { fastPath: true });
  if (found.kind !== "found") return [];
  try { return (JSON.parse(found.text).memories ?? []).filter(usable); }
  catch { return []; }   // a bad model turn must never fail the job
}

// Version B, the same leniency, but the return value says which situation it was.
function parseNew(text, { guard = true, fastPath = false } = {}) {
  const no = (outcome) => ({ outcome, understood: false, candidates: [] });
  if (!text.trim()) return no("empty_response");
  const found = locate(text, { fastPath });
  if (found.kind === "absent") return no("no_json_found");
  if (found.kind === "unbalanced") return no("unbalanced_json");
  let parsed;
  try { parsed = JSON.parse(found.text); } catch { return no("malformed_json"); }
  if (!Array.isArray(parsed.memories)) return no("missing_memories_key");
  const raw = parsed.memories;
  const entries = guard ? raw.filter((e) => typeof e === "object" && e !== null) : raw;
  const candidates = entries.filter(usable);   // throws on a null entry when the guard is off
  if (candidates.length) return { outcome: "ok", understood: true, candidates };
  return { outcome: raw.length === 0 ? "empty_result" : "all_candidates_invalid", understood: true, candidates };
}

console.log("1. the old parser on nine answers:");
const empties = ANSWERS.filter(([, a]) => parseOld(a).length === 0).length;
for (const [name, a] of ANSWERS) console.log(`   ${pad(name, 25)} ${parseOld(a).length === 0 ? "[]" : parseOld(a).length + " candidate"}`);
console.log(`   ${empties} of ${ANSWERS.length} answers came back as [].`);

console.log("\n2. the same answers through the version that names the outcome:");
console.log(`   ${pad("answer", 25)} ${pad("old", 14)} ${pad("outcome", 24)} understood`);
const seen = new Set();
let unreadable = 0;
for (const [name, a] of ANSWERS) {
  const r = parseNew(a), old = parseOld(a);
  if (old.length === 0) { seen.add(r.outcome); if (!r.understood) unreadable++; }
  console.log(`   ${pad(name, 25)} ${pad(old.length ? old.length + " candidate" : "[]", 14)} ${pad(r.outcome, 24)} ${r.understood ? "yes" : "no"}`);
}
console.log(`   ${empties} empty arrays, ${seen.size} different outcomes, ${unreadable} of them answers that could not be read.`);

console.log("\n3. a shortcut that tests the ends of the reply, and not whether the braces balance:");
const CUTS = [
  ["cut off after a nested object closes", '{"memories":[{"kind":"fact","text":"a"}'],
  ["cut off inside a string",              '{"memories":[{"kind":"fact","te'],
  ["a complete reply",                     '{"memories":[{"kind":"fact","text":"a"}]}'],
];
console.log(`   ${pad("input", 40)} ${pad("with the shortcut", 19)} balance scan only`);
for (const [name, a] of CUTS) console.log(`   ${pad(name, 40)} ${pad(parseNew(a, { fastPath: true }).outcome, 19)} ${parseNew(a).outcome}`);

console.log("\n4. two things about the failure paths:");
try { parseNew('{"memories":[null]}', { guard: false }); console.log("   a null entry, no guard: no error"); }
catch (e) { console.log(`   a null entry with no guard: threw ${e.name}: ${e.message}`); }
console.log(`   a null entry with the guard: ${parseNew('{"memories":[null]}').outcome}`);
for (const text of ['{"memories":[alice@example.com]}', '{"memories":[{"text":"alice@example.com" oops}]}']) {
  try { JSON.parse(text); } catch (e) { console.log(`   JSON.parse message: ${e.message}`); }
}

console.log("\n5. two more ways an outcome hides inside another one:");
const sanitize = (xs) => xs.filter(usable), score = (xs) => xs.filter((c) => c.text.length > 3);
for (const [name, raw] of [["the sanitiser drops every entry", [{ text: " " }]], ["the scorer drops every entry", [{ text: "hi" }]]]) {
  const sanitized = sanitize(raw), scored = score(sanitized);
  console.log(`   ${pad(name, 33)} count after scoring: ${scored.length}   count after sanitising: ${sanitized.length}`);
}
const first = { run: { rejected: 2 } }, patch = { run: { outcome: "ok" } };
const shallow = { ...first, ...patch };
const nested = { ...first, run: { ...first.run, ...patch.run } };
console.log(`   receipt written over the first attempt's:  ${JSON.stringify(shallow)}`);
console.log(`   receipt merged one level down:             ${JSON.stringify(nested)}`);

Output from Node 22.22.0 on Linux. The run on Node 22.22.2 printed the same text.

1. the old parser on nine answers:
   found nothing             []
   empty reply               []
   prose, no object          []
   cut off in the object     []
   malformed JSON            []
   valid JSON, wrong shape   []
   only blank entries        []
   a null entry              []
   one usable fact           1 candidate
   8 of 9 answers came back as [].

2. the same answers through the version that names the outcome:
   answer                    old            outcome                  understood
   found nothing             []             empty_result             yes
   empty reply               []             empty_response           no
   prose, no object          []             no_json_found            no
   cut off in the object     []             unbalanced_json          no
   malformed JSON            []             malformed_json           no
   valid JSON, wrong shape   []             missing_memories_key     no
   only blank entries        []             all_candidates_invalid   yes
   a null entry              []             all_candidates_invalid   yes
   one usable fact           1 candidate    ok                       yes
   8 empty arrays, 7 different outcomes, 5 of them answers that could not be read.

3. a shortcut that tests the ends of the reply, and not whether the braces balance:
   input                                    with the shortcut   balance scan only
   cut off after a nested object closes     malformed_json      unbalanced_json
   cut off inside a string                  unbalanced_json     unbalanced_json
   a complete reply                         ok                  ok

4. two things about the failure paths:
   a null entry with no guard: threw TypeError: Cannot read properties of null (reading 'text')
   a null entry with the guard: all_candidates_invalid
   JSON.parse message: Unexpected token 'a', ..."emories":[alice@exam"... is not valid JSON
   JSON.parse message: Expected ',' or '}' after property value in JSON at position 41 (line 1 column 42)

5. two more ways an outcome hides inside another one:
   the sanitiser drops every entry   count after scoring: 0   count after sanitising: 0
   the scorer drops every entry      count after scoring: 0   count after sanitising: 1
   receipt written over the first attempt's:  {"run":{"outcome":"ok"}}
   receipt merged one level down:             {"run":{"rejected":2,"outcome":"ok"}}

What it shows, with its limits. Section 1: with the old shape, eight of nine answers return the same empty array, whatever the reason. Section 2: the same nine answers get eight outcomes, and the eight empty arrays carry seven of them, because two answers share one. Five of the eight empty arrays are answers that could not be read. Section 3: a shortcut that tests the ends of a reply labels a reply cut off after a nested object closes as malformed, and a balance scan alone labels it unbalanced. Section 4: with no filter, a null entry throws a TypeError, and with the filter it becomes an answer that was read and held nothing usable. Of the two parse messages, one contains a slice of the input and the other only a position, so in these runs whether a message held input depended on the kind of error. Section 5: counting after scoring cannot tell a scorer rejection from a clean-up rejection, and a write that replaces a nested object drops the first attempt's key. It does not show how often any model's answers fall into any of these classes, how Northsun's parser behaves today on any input, or the SQL behaviour from round two.

How to check your own parser

Five minutes, in order. Search your code for the places that turn a failure into an empty value: return [], ?? [], a catch that returns nothing. For each, list the situations that reach it, and ask whether the caller can tell them apart and whether any of them is counted or logged. Feed the function the kinds of answer in the script's table, including a truncated one and one with a null entry, and compare what comes back. If you need the leniency, keep it and return the outcome too. Store the outcome next to the event and not the content. And read what your success status really means: whether it still says success when the answer could not be read.

What this still does not prove

• The script is not Northsun's code. Its toy parsers stand in for the pattern, and its output says nothing about how any Northsun code behaves today.
• This is reported from commit messages, a runbook, a test file, the old and the merged parser's source, the types and pipeline code, the main branch's storage code, a build-plan row, a peer-review note and private tracker notes. Apart from the check of the two parsers described above, none of it was re-run, the repository is private, and I found no deploy record, so the post claims a merge and nothing more.
• Nothing here says that facts were lost, that none were, or how often a model's answer is unreadable. The record says that for the two events it cannot be known.
• The privacy boundary is as the runbook and its test describe it. It is not a security claim about any system.

FAQ

Why not make the parser strict and fail the job?

The runbook records the decision and gives reasons: the raw observation is already stored, so declining to retry destroys nothing; it treats extraction at temperature zero on an identical prompt as repeatable, so a retry repeats essentially the same computation and is billed again; and a retry that discards the output again is not a repair. What the fix changed was visibility, so that an operator can decide what to do about an unreadable answer.

What is the difference between empty_result and all_candidates_invalid?

empty_result is the model's own answer: the list was empty. all_candidates_invalid means the model offered entries and the parser's clean-up rejected every one: blank or textless entries, or entries that were not objects. The runbook reads that as the parser's side and not the model's, and the receipt's count of entries offered against entries kept shows how many were dropped.

Does this mean facts were lost?

This post does not say so, and the record says it cannot be known for the two events. What the merged change is meant to change is whether the next one can be classified from a stored row, without replaying anything, once the code is running: the answer was understood, or loss was possible. For an unreadable answer the receipt cannot say what the answer held, because it stores no model text, and the runbook's way to find out is to replay the stored transcript.

Is it deployed?

I found no record of a deploy or a production check, so I say it is merged. The status vocabulary follow-up was also still open on 7 October 2026.

The receipts

The Northsun repository and issue tracker are private, so these are paths and commits for anyone with access to check. The tracker notes are summarised, not quoted at length, and they carry internal figures, which this post leaves out.

For other stories where success was the default, read four more ways a check goes green without running, the analytics call that couldn't fail and how 29 review rounds hardened a CI gate that nothing ran.