A reviewer does not ask whether the model was right. A reviewer asks what the system did on the cases where it was unsure, and whether anyone can tell afterwards. Most of the answer lives outside the model call, in code that runs before it and code that runs after it.
Everything below runs today in Northmark’s own federal opportunity pipeline. That pipeline is an internal operating system: it is not a product, it is not priced, it is not demonstrated, and it is not deployed anywhere outside Northmark. It is described here because a running mechanism is checkable in a way a diagram is not. Python, LangChain and LangGraph, PostgreSQL, GitHub Actions, containerised services.
Validation runs before the model, not after it
The common shape is: call the model, then check what came back. It is easy to build and it is the wrong order. Checking after the call means the model has already seen a record that should never have reached it, and the only artifact of that is a response somebody now has to reason about.
In Northmark’s pipeline the checks are deterministic and they run first. Schema validation — is the record shaped the way the model contract says. Rule validation — does the record satisfy the conditions this path assumes. Eligibility validation — is this a record the system is permitted to act on at all. A record that fails any of the three never reaches a language model.
Two properties fall out of that ordering, and both matter to a reviewer. Every rejection has a named rule attached, because a deterministic check knows why it said no. And the population that reached the model is a defined population, which is what makes any later statement about model behaviour a statement about something rather than about everything.
The test for whether your validation is really deterministic: run the same record twice with the model unavailable. If the two runs disagree, or if either one gets further than the first failed check, the validation is advisory and the model is the gate.
Exactly two things the system may close by itself
Autonomy is enumerated, not inferred. In this pipeline there are exactly two mechanical reasons a case may be closed without a person: both are conditions where the outside world has already decided, and no judgment is being exercised. Everything else — every case the system has an opinion about — arrives as a recommendation with a person’s decision attached.
Two is not a modest number chosen for optics. It is the count of cases where closing is mechanical. A short list is the property that makes the boundary reviewable: a reader can hold two rules in their head, check them against the code, and know the shape of what the system is allowed to do. A list of forty autonomous actions is not a boundary, it is a policy nobody has read.
The design question this answers is the one every AI review eventually reaches: what does it do without asking? An enumerated list answers it in a sentence. A confidence threshold does not, because a threshold is a claim about the model’s calibration and the reviewer has no way to check that claim.
A close missing its reason code raises instead of completing
Closing a case requires two things: a reason drawn from a fixed enumeration, and a written lesson. If either is absent the close raises an exception. It does not log a warning, it does not close with a null reason, and it does not fall back to a generic code.
That choice is deliberate and it is worth naming why, because the alternative is what almost every system does. A close that completes with a missing reason produces a database that looks complete and is not. Six months later the reason column is 30% null, nobody knows which 30%, and the reporting built on top of it was wrong the whole time. Raising is loud, it is annoying, and it keeps the record honest at the point where honesty is cheap.
Reviewer notes are append-only. A note is never edited and never deleted, so the sequence of what somebody believed and when is recoverable. Correcting a judgment means adding the correction, not replacing the judgment — which is the same discipline a case file has always had, for the same reason.
One outbound choke point, and every refusal logged with its cause
Anything the system sends outside itself goes through a single function. Not a convention, not a code-review habit — one path, so that the question what did this system emit has an answer that is a file rather than an investigation.
The choke point refuses, and refusals are logged with their cause. That log is the artifact worth more than the send log. A count of what went out tells a reviewer that the system works. A count of what was stopped, with the reason attached, tells them the gate is doing something — and a gate that has never refused anything is a gate nobody has tested.
Open items carry follow-up dates, swept on a schedule. An item that needs a person is a commitment, and a commitment without a date is a thing that quietly does not happen.
A run that did not do its work reports failure
Scheduled jobs fail silently by default. A connector returns an empty page, the loop iterates zero times, the job exits zero, and the dashboard is green. Nobody finds out until somebody notices the numbers stopped moving, which on a weekly cadence is weeks.
Every run carries health assertions that fail the run rather than reporting a success that did not happen. The rule underneath, written down and enforced rather than remembered: a run that produced no work must not exit as though it did. If the assertion cannot distinguish a genuinely empty day from a broken connector, the assertion is not finished.
This is the cheapest gate in the list and the one most often missing. It costs a few lines per job and it is the difference between finding a broken ingestion on the day it breaks and finding it in a quarterly review.
What the completeness floor caught when it went red
The pipeline holds a completeness floor as a stated target: 90%. It reported 89.5% for several days. The number is on this page because of what happened next, which is the only part that tells a reader anything: the floor was not lowered, and the backfill everyone kept running was a mop. The cause was upstream — a bid decision could be recorded against an opportunity whose description text had never been fetched, which is a decision recorded against a title. A single sanctioned constructor now refuses that row and logs the refusal with its cause. The ratio reports 1.000 across all 46 rows today.
A threshold that has never been breached is not evidence of quality, it is evidence that the threshold was set below the observed value. The useful property of a floor is that it is capable of reporting a miss, out loud, to the person who set it — and a reader can tell the difference between a measure that is watched and a measure that is decoration by whether it is ever red.
In a federal delivery review the same distinction applies to whatever quality gate a program has written into a contract. Ask when it last failed. If the answer is never, the next question is what the gate is measuring.
Not built here: precedence rules, anomaly detection, role-based access
The gates above are the whole list. Several mechanisms a reader might reasonably expect alongside them are absent from this system: source-of-truth precedence rules, anomaly detection, role-based access control, prompt and model version control, owner assignment on open items, automatic re-test of an original artifact, and any scenario or what-if comparison. There is also no user interface, no guided search and no export inside the pipeline itself.
Some of those are real gaps and some are scope. Version control on prompts is a real gap in any system that will be reviewed, and it is a gap here. Role-based access is scope: there is one operator, and access control on a single-operator system is theatre until there is a second person.
The distinction between the two is the thing worth carrying into someone else’s design review. A mechanism that is absent because the system is small will need to exist before the system is not. A mechanism that is absent because nobody thought of it is found by a reviewer, not by the team.
What this is not
None of this constitutes an authorization. Northmark holds no ATO, no RMF package at any level, and nothing under FedRAMP or CMMC. A deterministic validation layer is an engineering property of a system; an authorization is a decision an authorizing official makes about a system in an environment, and the two are not substitutes.
The pipeline described here is Northmark’s own. It is not offered, sold, demonstrated or deployed anywhere else, and there is no version of this document that ends in a product.
Ask about anything in this guide.
The methods here are the ones we use. If one of them is wrong for your program, that is worth a conversation.