Model governance proof trail
Defend a model-driven decision to the examiner who asks eighteen months later.
A regulated firm publishes a marketing claim. An ML model output feeds the call to substantiate it. Months later, internal audit, model validation, and an examiner want to know why that output, on that date, was justified. This page walks one such decision, end to end, in synthetic form: the messy evidence in, the proof trail out, and what an outside reviewer would need to check offline.
The scenario.
Meridian Ridge Asset Management (synthetic) runs a marketing-review workflow. Before a performance claim goes out, an ML peer-ranking model scores where a strategy sits against its peer group, and that output feeds the decision to substantiate the claim or hold it for a human. The claim under review reads: the Meridian Ridge Quantitative Equity strategy achieved top-decile risk-adjusted returns among its peer group over the trailing five years.
The honest reviewer of that decision sits outside the desk that made it. Internal audit tests the control. Model validation, as a second line, has to re-derive the output. An examiner, under the firm's own obligations, can ask for the basis of the claim long after the model, the reference data, and the reviewer have all moved on. The firm applies its own regime here: the SEC Marketing Rule expectation to substantiate a claim on demand, and model-risk governance in the SR 26-2 (successor to SR 11-7) tradition. Those are the firm's obligations, not a Forge credential. Forge supports that documentation pattern once a workflow is qualified. This synthetic page does not produce evidence for a real control, certify compliance, or replace the firm's judgment.
| The decision | Given the model output and the firm's encoded marketing-review policy, substantiate the claim for distribution, hold it for human review, or do not substantiate it. |
|---|---|
| Control depth | Illustrative evaluate pattern only. The page shows how evidence is compared with an encoded policy. It calls no API and is not a network or execution control. |
| What is pinned | The model version (peer_ranking_model v4.2.1), the policy version and its hash, a substantiation confidence threshold, a source-reliability floor, and a data-freshness window. All bound into the proof trail. |
| Who re-derives it later | Internal audit, model validation, and the examiner. In a real managed deployment they would need the complete exported record plus a full signing-key ID provisioned through a separate trusted channel; this synthetic page provides neither. |
The messy evidence in.
These synthetic artifacts illustrate the input shape. In a real integration, any extraction would run before the checked path and each normalized value would need a cited source and content hash. The illustrative weakness is visible before a rule runs: one peer-universe snapshot is older than the encoded freshness window.
Peer-ranking model output
peer_ranking_model v4.2.1 returns a trailing five-year peer percentile of 8, inside the top decile. Normalized outside the signed path.
Feature vector fed to the model
Sixty months of strategy and benchmark return series plus the peer universe identifier that the model scored.
Peer universe reference (stale)
The peer set the percentile is measured against. This snapshot is older than the policy freshness window, which is what caps confidence.
Risk-adjusted-return methodology
The approved memo defining how risk-adjusted return is computed for the claim, on a Sharpe basis.
Read the shape of it before any policy runs: the model leans toward substantiate, but the peer set it depends on is stale, and the whole claim rests on a reference snapshot no one refreshed. The point of the run is to make that weakness explicit, attributed, and reproducible, rather than to smooth it over.
The proof trail out.
This illustrates the managed artifact: the customer's policy and model version are recorded alongside a semantic replay basis. Under a pinned engine and configuration, replay should reproduce the governed action and replay key; a newly issued record still has its own record ID and signature metadata. Fields resemble DecisionRecord concepts, but every value below is illustrative and no record is signed by this page.
The policy, applied to these exact inputs, does not clear the claim on its own. The leading substantive outcome is below the auto-substantiate threshold and one substantiation source falls below the reliability floor, so the record routes to a human. This is not "the model said yes."
The leading outcome is SUBSTANTIATE at 0.71. The policy requires 0.80 to clear a claim without a human, so the selected_action is the hold, not the argmax. The posterior is shown, not summarized away, so the reviewer sees exactly how close the call was.
- Stale peer universe (e3). The percentile is measured against a snapshot older than the freshness window. Confidence is capped pending refresh. If refreshed peer data moves the strategy out of the top decile, the substantive outcome flips and the claim cannot be substantiated as written.
- Leading outcome below threshold. SUBSTANTIATE sits at posterior 0.71, under the 0.80 auto-substantiate bar. The margin is recorded, not rounded up.
The model-validation sign-off boundary is explicit, not implied.
- Forge does not make the call. The illustrative record shape carries a disposition and human boundary. It shows PENDING with a proposed role route because the synthetic policy trips its confidence gate and reliability floor. No routing occurs.
- Model validation, as a second line, re-derives the output. The workflow lets a supervisory principal sign the claim off or hold it. Refreshed evidence or a later human disposition should become a new event referencing the original issuance, not an edit to it. This page creates no issuance or amendment.
- This is calibration, not training. A qualified workflow requires an authorized customer outcome and explicit retention opt-in. With no retention opt-in, no calibration row is written. Decision, audit, idempotency, and accountability-record retention is governed separately by the deployment and contract. Replay separately compares the governed action and replay key under the pinned engine and configuration; it does not certify the input data.
- The reviewer who must be satisfied sits outside the desk. Internal audit, model validation, and the examiner all need to trust a decision they did not make. If the only account of why the claim was justified comes from the same team that produced it, that is not independence. You cannot build your own independence.
What real offline verification would require.
This page is not a verification package and the values above cannot be verified. In a managed deployment, an external reviewer would receive a complete signed v2 record plus the full expected signing-key ID through a separate trusted channel. The key embedded in a record, or returned by the same API, cannot establish Forge attribution on its own. Replay is a separate check of the governed action and replay key under the pinned engine and configuration.
$ python3 scripts/forge_verify_record.py record.json \
--expect-key-id ed25519:<full-sha256-key-id-from-trusted-channel>
Expected for an authentic, untampered v2 record:
complete-record signature ........................ VALID
derived key ID ................................... MATCH
Then replay separately with the pinned engine/configuration and compare:
selected_action .................................. MATCH
replay_key ....................................... MATCH
This synthetic page creates neither record.json nor a verification result.
Integrity plus independently pinned attribution.
A valid complete-record Ed25519 signature detects changes to delivered decision fields. Attribution to Forge additionally requires the verifier to match the key derived from the signature's public-key bytes to the full expected key ID provisioned through a separate trusted channel.
Replay separately checks the governed result.
The reviewer separately re-runs the recorded basis with the pinned engine, model, configuration, and policy, then compares selected_action and replay_key. Replay does not promise a byte-identical response envelope or reuse of record identity, issuance time, or signature metadata.
| What a green managed verification can establish | The complete delivered record verifies under the independently pinned signing identity, and a separate replay matches the governed action and replay key under the pinned engine and configuration. This page itself establishes none of those results. |
|---|---|
| What verification does not prove | That the 0.80 threshold, the reliability floor, the top-decile claim, source data, policy, or business judgment is correct. It does not establish a legal or regulatory conclusion. Those judgments stay with the firm, its counsel, and its examiners. |
What a proof trail contains.
The illustrative fields below show the v2 review shape. A record contains the fields applicable to the validated request and disposition; optional or unevaluated safeguards are not asserted as present.
- record_id A stable identifier for this decision record.
- selected_action The disposition the policy produced. Substantiate, hold for human review, or do not substantiate.
- confidence The decision confidence, tied to the policy threshold that governs whether a human is required.
- posterior The distribution over the candidate substantive outcomes, shown in full rather than reduced to a single number.
- evidence_chain Each item carried with its source and a truncated sha256 content hash, including the model output and version, the input features, and the substantiation sources.
- uncertainty The held uncertainty, attributed to specific evidence, with the sensitivity that would flip the outcome.
- human_review boundary The boundary, the status, the role the record routed to, the reason, and the note that the human keeps the decision.
- replay_key The semantic handle used to compare the governed action and replay key under the pinned engine and configuration. A new issuance receives its own record ID and signature metadata.
- complete_record_signature and key_id A complete-record Ed25519 signature plus a key ID that must match the verifier's independently provisioned full trust anchor.
Start with one decision surface.
You do not have to re-plumb model governance to see whether this holds up. Pick one high-consequence surface, a marketing-claim substantiation, a model-change approval, or an agentic override that no human currently logs. We review the policy and map one sanitized or synthetic decision to the engine. A managed deployment and any independently verifiable artifact remain conditional on completing the published security, audit, build, deployment, and operational gates. Forge Orbital is at design-partner stage.