Obligation Ledger
Air-gapped retrieval where the evidence check is code
A local contract-obligation service where every extracted value must quote its source page, the quote is verified in Python, and every derived date is discarded from the model and recomputed.
- Tests
- 128
- including the hallucinating-model ablation
- Ablation result
- approval blocked
- with a hallucinating model, verification catches the invented date
- Retrieval on the lease fixture
- 7 of 7
- material clauses retrieved by lexical search alone
- Network calls leaving the machine
- 0
- weights loaded locally only; one outbound connection, to the local model
Figures with a dotted underline carry their source. Hover one to see whether it was verified in code or reported in project documentation.
Context
The situation
Contract obligations — renewal dates, notice windows, escalation caps — are buried in prose and in filled form templates. Missing one has a direct financial consequence, which is exactly why nobody trusts an unaudited model to find them. The build had a hard constraint on top: a one-day hackathon where everything had to run fully offline on a single Dell Pro Max with an NVIDIA GB10, with no cloud inference of any kind.
The problem
Retrieval-augmented extraction fails in a specific way: the model returns a plausible value with a plausible-looking quote, and nothing checks that the quote exists, that it is on the page claimed, or that the arithmetic derived from it is right. The output is confident and unfalsifiable.
What I built
Make each of those checks a piece of code with a test. Verification normalises the text and matches the claimed quote against the claimed page, failing a quote that is real but located elsewhere. Date arithmetic is removed from the model entirely and recomputed from verified inputs. And an adversarial model adapter that deliberately hallucinates is wired into the test suite, so the guardrail is proven rather than asserted.
A model reading a contract will confidently produce a notice deadline, and if it is wrong someone misses a window. This service is built on the bet that the engineering around the model is what makes its output trustworthy. Three rules are enforced in code rather than in interface copy: every value must quote the contract and that quote is verified against the page it claims; every derived date is discarded on principle and recomputed in Python with its formula stored alongside; and the service never commits anything, returning a hard "cannot approve" whenever any field fails. It runs fully air-gapped against a locally served open-weights model, on a single on-premise box.
Architecture
How it fits together
- Contract PDFinput
page numbers preserved
- Page-aware chunkingdeterministic
offsets and overlap
- Hybrid retrievaldeterministic
BM25 + local vectors, fused by rank
- Local modellanguage model
proposes values and quotes
- Quote verificationguard
matched against the claimed page
- Date arithmeticguard
recomputed in code, never trusted
- Approval gateoutput
blocked if any field fails
Key decisions
The choices that shaped it
- 01
A real quote on the wrong page fails
Verification normalises Unicode punctuation, ligatures, non-breaking spaces and soft hyphens, then tries exact substring match against the claimed page, falling back to a bounded fuzzy match anchored on the longest common block and capped by both a similarity threshold and a length ratio. A quote that exists elsewhere in the document fails, with a reason naming the page it was actually found on.
verification.py
- 02
Filled form templates break the naive check
A lease template expresses values as ticked options, so “☐ Rent will NOT be increased” is a genuine quote of a value that was not agreed. A naive "the quote is in the document" check passes it. Verification resolves the nearest checkbox governing the quoted span and fails an unticked one.
- 03
The product says QUOTE MATCHED, not VERIFIED
The status a matched value carries is deliberately narrow, and the legend sits in the interface rather than in the pitch: a matched quote means the value appears in the text it cited. It does not mean the model cited the right clause. That distinction is the difference between a check that can be trusted and one that quietly overclaims, and putting it in front of the reviewer rather than in a footnote is what makes the approval step meaningful.
- 04
The model never verifies its own quote
Asking a model to check its own output is asking the same failure mode to audit itself. Verification is deterministic Python with no model in the loop, which is also what makes it testable.
- 05
Reciprocal rank fusion, for a stated reason
BM25 scores and cosine similarities live on different, document-dependent scales, so any fixed weighting between them is a guess. Ranks are directly comparable, so the two retrievers are fused by rank rather than by score.
retrieval.py
- 06
Arithmetic leaves the model entirely
The model reads a term-end date and a notice duration off the page; Python computes the deadline. Month arithmetic clamps to short months, and every computed value stores its formula and the ids of the verified obligations it was derived from.
date_math.py
- 07
An adversarial adapter proves the guardrail
A model adapter that deliberately invents values is part of the test suite. The ablation asserts that verification catches the invented date and that approval is blocked. A guardrail with no failing case is a claim, not a control.
- 08
Degrade loudly, not silently
If the embedding model fails to load, the service logs and falls back to lexical retrieval rather than failing outright — which means the absence of errors proves nothing. The health endpoint reports which retrieval mode is actually active, and that is the thing to trust.
Evidence
What it actually looked like
The status is deliberately narrow, and the legend is in the product rather than in the pitch. The contract is a synthetic fixture.
Interface
The four of us, and the machine. That box in the middle is the Dell Pro Max with GB10 — every model call, every embedding and every byte of the register stayed on it.
HardwareMeasured
Every figure, with its source
- Tests
- 128
- including the hallucinating-model ablation
- Ablation result
- approval blocked
- with a hallucinating model, verification catches the invented date
- Retrieval on the lease fixture
- 7 of 7
- material clauses retrieved by lexical search alone
- Network calls leaving the machine
- 0
- weights loaded locally only; one outbound connection, to the local model
- Obligation types
- 11
Figures with a dotted underline carry their source. Hover one to see whether it was verified in code or reported in project documentation.
My contribution
Sole author of the retrieval and extraction service. Three teammates owned the frontend, the main application backend, the dataset and the local model runtime.
What I owned
- PDF ingestion with page numbers preserved, and page-aware chunking with offsets
- Hybrid retrieval: lexical BM25 and local dense vectors fused by reciprocal rank fusion
- Structured obligation extraction against a typed schema shared with the main backend
- Deterministic source-evidence verification, including the filled-form checkbox case
- Deterministic date arithmetic with stored formulas and input provenance
- Real, fake and deliberately-hallucinating model adapters, and the ablation test built on the last one
- Two integration paths — an HTTP service and an in-process module contract — sharing one pipeline
What I did not
- The frontend and the approval interface
- The main application backend, the committed ledger and its hash chain
- The local model runtime and the contract dataset
Stack and limits
Built with
- Python 3.12
- FastAPI
- Pydantic
- PyMuPDF
- sentence-transformers
- BM25
- SQLite
- NumPy
- Docker (multi-arch)
- pytest
- ruff
Techniques
- Hybrid lexical + dense retrieval
- Reciprocal rank fusion
- Deterministic evidence verification
- Adversarial model adapters for guardrail testing
- Air-gapped local inference
What it is not
- One component of a four-person product. I did not build the interface, the committed ledger or the model runtime.
- Evaluated on two synthetic fixtures — a prose services agreement and a filled lease form — not on a real contract corpus.
- Scanned documents with no text layer are rejected rather than OCR’d. Out of scope by design.
- Ambiguous day-first versus month-first dates are read as US format, which is a decision worth revisiting against real data.
What carried forward
The pattern that keeps recurring: the model is good at reading and terrible at being accountable, so give it the reading and keep the accountability in code. Every rule here is a function with a test, and the adversarial adapter is what turns "we verify quotes" from a sentence in a README into something a reviewer can run.
Next
Chapter 02
Forkcast
A language model can explain a decision. It cannot be allowed to invent the numbers behind it. Forkcast enforces that in code, and exposes the verifier so anyone can attack it.
Explore case study→July – August 2026
Omnivision
Assistive tools describe scenes but carry no memory, and cannot tell recognising something from being confident about it. I built the part that refuses.
Read case study→