The Resume Validator

(draft — findings as of 2026-08-12; the experiment is ongoing)

The problem

Whatever else a generated resume gets wrong, grounding is the question that has to be answered correctly: does the document claim anything the candidate's record does not support? A resume that reads beautifully and fabricates is worse than useless. The calibration work ended with an uncomfortable verdict on this front — no configuration of holistic judge could be trusted to gate or rank pipelines — so the grounding check was rebuilt on the opposite philosophy: stop asking judges holistic questions. Break the resume into very small questions, ask each one separately, and fence every answer with a deterministic check.

On its first real test the rebuilt validator went nine for nine. Every known defect was caught with a verbatim citation, and every clean line was left alone. This page covers the design, the failures on the way to it, and what the result does and does not establish.

The test material

The defects were not invented for the test; they are lines the real generation pipeline actually produced, each representing an organic failure mode of LLM resume writing:

• Actor inflation — a bullet claiming "Led end-to-end development of CLEAR" where the source record grants product ownership of changes (writing requirements, grooming, validating) but not the build.
• Category drift — a portal the source data calls a "client-facing web portal", promoted by the resume to a "SaaS portal".
• Misattached structure — a custom MCP server attached to the wrong parent system.
• Tense promotion — work the record says is "being added", claimed as "implemented".

Each defective line is paired with a clean twin line to catch false alarms, giving nine scored lines across three resumes and one candidate record.

Why the first validator lost

The first attempt tried to make grounding checkable by structuring it: extract subject–predicate-style assertion tuples from both documents — actor, relation, scope, modality, each with its own evidence span — bind the entities, and compare with deterministic rules. The appeal was that nothing subjective survived to the comparison step. The cost was that everything subjective got pushed into extraction, where it failed quietly: one honest proposition would shatter into partial tuples whose axes disagreed, and a sentence copied nearly verbatim from the source could still read as an overstatement. The comparator did exactly what it was told, with inputs that could not carry the meaning.

The fact-verification literature had already been through this. The designs that work — decompose-then-verify, in the FActScore lineage — do not ask models to emit schema. They ask for atomic claims in plain language, then judge each claim alone against the full source, with the structure living in the judge's instructions rather than in the data format. The validator is that design, plus one addition the literature underuses: every model output that can be checked by string matching, is.

The design

The pipeline has three stages. Models make every judgment call; code checks every checkable artifact.

1. Decompose (model, fenced by code). Each resume line is restated as self-contained declarative sentences, one fact each, preserving exactly the strength of the original: who acted, how far the work got, how much it covered, every hedge. Each claim carries a verbatim quote from the line, and code verifies the quote character-for-character. A claim whose quote is not actually in the resume is set aside visibly rather than silently trusted or dropped.

2. Verify (model, fenced by code). Each surviving claim is judged alone against the entire candidate record, with three verdicts that are not allowed to blur: supported (the record communicates it), contradicted (the record establishes something incompatible), and unsupported (the record is silent). Silence is not conflict, and silence is not support. Supported and contradicted verdicts must quote the record verbatim, which code again checks. The axis discipline the old comparator tried to enforce in schema lives in the judge's rubric instead: actor, status, extent, lifecycle, attribution, identity, structure.

3. Score (code). A defect counts as caught only when a non-supported verdict lands on a claim quoting the defective span; being wrong loudly somewhere else on the line does not count. A clean line is scored on whether it was accused at all.

The principle throughout is that judgment stays with the model, but lying gets hard. A model cannot invent a quote, attach evidence to a verdict that should carry none, or quietly skip a line, without the fence turning the failure into a visible, classifiable artifact.

The cheap runs

The first two runs put Haiku 4.5 on both stages, at roughly sixty cents per full pass. They scored 7/9 and 6/9, and the misses were worth more than the scores, because the fences said exactly where the losses were:

• The judge's verdicts were mostly right; its paperwork was sloppy. About a quarter of verdicts initially failed the evidence check — not because evidence was invented, but because the model stitched real quotes together with ellipses, or attached a "nearest miss" quote to verdicts that were supposed to carry none. Loosening the checker to verify each stitched fragment verbatim recovered these; invention still blocks, and shape problems stay visible on the row.
• Decomposition was the real bottleneck. The SaaS-portal line was missed in both runs for the same reason: the category label got folded into a long claim about serving as a conduit between teams, so the category question was never put to the judge. The clean modality line false-alarmed in both runs for the mirror-image reason: "being added to enable a governed flow" got strengthened into "the additions enable," and the judge then correctly contradicted a claim the resume never made. Targeted prompt rules — a label used in passing is its own claim; purpose is not achievement — changed nothing on either line.
• One judge miss was recall, not judgment. A clean ownership line was accused because the judge reasoned carefully from the wrong part of the record, missing the sentence that settles the question. The same claim passed in the other run, so single-chain verdicts at this model tier are noisy at the boundaries.

The Terra run

Swapping the decomposition model to GPT-5.6 Terra settled the prompt-versus-model question immediately: 511 claims from all 72 lines, zero quote violations, zero skipped lines, and both chronic decomposition failures gone. The category claim finally exists as its own sentence ("The SpectrumEnterprise.net portal is a SaaS portal"), and the purpose clauses stay hedged ("being added to enable a multi-step, governed flow"). This was the same prompt Haiku had been given. The rules were fine; the model reading them wasn't.

With Terra also judging its own claims, the scorecard filled in:

506 of 511 verdicts carried evidence that survived the verbatim check — about a one percent loss, versus four to seven percent under Haiku. The catch that had been structurally impossible in every earlier run reads like this:

Claim: "The SpectrumEnterprise.net portal is a SaaS portal." Verdict: contradicted. Evidence: "developing the SpectrumEnterprise.net client-facing web portal." The source explicitly categorizes SpectrumEnterprise.net as a client-facing web portal, not a SaaS portal. The identity axis is incompatible with the claim's SaaS classification.

That is the shape of answer the holistic judges never produced: a specific claim, a specific verdict, a verbatim receipt, and a stated reason on a named axis — every piece of which a program checked before a human saw it.

What this shows, and what it doesn't yet

The through-line from the calibration work holds up well. The same class of model that reads well and scores badly when handed a whole resume performs decisively when the question shrinks to is this one sentence supported by this one document and the answer is fenced by string checks. The judgment didn't get better; the questions got smaller and the lying got harder.

What it does not yet show is behavior in the wild. Nine known defects across three resumes is an instrument check, not a field trial. The organic calibration cases — the mistargeted case and the rest of the set where the holistic judges died — have not been through this pipeline yet, and they are the real test. Also open:

• The cost floor. Terra decomposition with a cheaper judge is untested. The failure analysis suggests decomposition needs the stronger model more than judging does, though the judge's one recall miss argues for some care.
• Multi-model chains. Whether any/majority/unanimous verdict policies across models buy real precision, or just averaging.

Findings will land here as the runs complete.