The Evaluator Reads Well and Scores Badly

I built a calibration set to find out whether an LLM judge could rank résumé-generation pipelines. It cannot, and for a while I read that as a failure. It was the wrong reading, because it measured the wrong output.

Across 288 judgments — sixteen cases, six prompts, three frontier judges — the numeric channel is close to inert. On the set's hardest case it is completely inert: eighteen configurations returned the same three numbers, and no prompt moved any of them by a point. The prose those same judges wrote is a different story. The diagnosis the scores could never reach is sitting in the rationale fields of the runs that got the score wrong.

Scores were never the deliverable. The evaluator feeds a collection step, and that step is where a résumé problem is supposed to become visible. So I ran it across all sixteen cases. It recovered the decisive diagnosis on the hardest case, verbatim and fully cited, for seven cents — and then showed two failure modes serious enough that it is not ready to run unattended. Both are in the wiring rather than the models.

The system this sits inside

A generation graph produces a résumé. A post-hoc evaluator graph (resume_eval/main/1) runs two independent rails over it:

• Deterministic lint — truncation, structure, candidate employment titles and dates. No model, no judgment.
• A qualitative review — six dimensions (basics, writing, argument, grounding, keywords, overall), each returning {judgment, rationale, evidence}, plus a summary and freeform comments.

A single review is not the product. Downstream, finding synthesis consumes one or more reviews and compiles a findings report. It receives only evaluator output — never the résumé, job description, or candidate data — so it organizes what was said without becoming a second judge. Its contract is written directly against the failure documented below:

Scan every supplied field, including summaries, rationales, evidence, and comments. A high numeric score does not cancel a problem described in prose.

Preserve the union of substantive findings. Do not require majority agreement and do not discard a finding because only one review instance mentions it.

Merge findings only when they diagnose the same underlying problem. An omitted detail is not the same diagnosis as a whole-document targeting failure.

Every finding must cite a review_id, a field name, and a verbatim quote. The caller re-checks each quote against the source text and rejects the entire synthesis if one fails.

(A separate veracity rail — claim decomposition and graph-based verification against candidate data — handles factual grounding. Out of scope here.)

The instrument

Sixteen cases. Eleven organic pipeline outputs and five synthetic near-twins. Each twin changes exactly one thing in a real résumé — a fabricated credential, a buried lead, inflated seniority, inline keyword bolding, a summary rewritten into imperative voice — so a paired comparison isolates whether the judge perceives that defect. The organic cases test the harder thing: recognizing a bad résumé when the failure is spread across the document rather than sitting in one sentence.

Owner-ratified labels. Score bands on a 0–2 scale, findings the review must mention, accusations it must not make. Twenty-two findings are graded hard, eight more soft, ten forbidden accusations test discipline. Three findings are weighted core — the ones that decide whether a case was really understood.

Six prompts. An anchored base rubric plus reading-instruction passages in a 2×2: mechanism (model reading is statistical rather than sequential, and can miss placement, order, and emphasis) and pressure (evaluators inflate, and must move the judgment when the rationale finds a real weakness). Two later prompts fold both into single integrated passages, one neutral and one deliberately harsh.

A claim check. A smaller model examined each review one claim at a time, every verdict backed by a quote: did it mention the required finding, did it make a forbidden accusation. 480 graded verdicts over the four-prompt matrix.

The scoring channel is inert

The posting behind the mistargeted case is not an engineering role. It is an embedded, business-facing AI job: discover workflows with business teams, prioritize use cases, coach users, drive adoption, leave a department more capable after rotating out.

The résumé answers a different job. Ten of its fourteen experience bullets are about systems built. None shows discovering a workflow with a business team; none shows coaching users. The summary asserts business enablement while the work history argues for an AI engineer. Not overweighted technical content — the wrong center of gravity, and exactly the judgment no linter can make.

The human band puts argument at 0. Every judgment in the study:

Eighteen configurations. Zero variance. The overall score is worse than a miss: the human band is 1, and Opus and GPT both return 1 — in band. They reach a defensible bottom line through an indefensible reading. A comparison that reads only overall sees nothing wrong at all.

It generalizes. Across 96 judgments per judge, overall barely moves:

GPT-5.4 never used the top score in 96 opportunities; excluding the one truncated case, 88% of its verdicts are the same integer. Band compliance says the same thing from another angle: 30 to 40 of 64 cells — 47% to 63% — for every judge under every prompt, no trend across the six. The direction of the misses is the one useful thing the scores produce: Gemini sits 25–30 cells above band against 3–6 below in every configuration; Opus is roughly symmetric; GPT trends harsher under pressure without becoming more perceptive.

Two scores do work. All eighteen configurations scored the truncated résumé overall 0, and all twelve claim-checked reviews named the truncation. And the grounding pair — a fake AWS certification, 6% inflated to 16%, 15× to 40× — was separated by every judge under every prompt, 18 of 18. Grounding is the one score I would gate on, because the defect is local, textual, and checkable.

The reading channel is live

Required-claim recall, each mention backed by a quote:

Opus and GPT made zero forbidden accusations in every configuration — ten opportunities each, none taken. Gemini made one or two per prompt. The same passages that leave the scores flat lift what the review says by fifteen to twenty points, and do not buy that recall with false positives.

What the union contains

If the deliverable is a findings report, the question is what an ensemble covers, not how good one review is. Coverage of the 22 hard-required findings:

Collection beats the best single review by nine points. Prompt variation is cheap diversity — four prompts against one model add four points, a second model adds five more.

Gemini contributes nothing. Its eight findings are all already covered by Opus or GPT, so it moves ensemble coverage by zero, while importing both of the study's false accusations: union semantics inherit every claim any member makes. Opus + GPT carries 0 of 10 forbidden accusations; adding Gemini carries 2.

And the core findings survive. The three core-weighted claims are exactly the whole-document failures the scores are blind to, and all three are in the union:

What the collection step actually produced

Containing a finding is not surfacing it, so I ran the synthesis on the mistargeted case's eighteen reviews — the four-prompt matrix plus the two later prompts, all three judges, including every run that graded the case as adequate.

Three configurations, and the first two are part of the result:

The cheap configuration failed in the most useful way. Its first finding cited cleanly; everything after it paraphrased quotes into something plausible that did not appear in any review. The validator caught all fifteen and rejected the whole report rather than emitting a partly-fabricated one. Fail-closed is the right behaviour, and it means the citation contract is doing real work rather than decorating the schema.

The configuration that passed returned this, third in its list, from reviews that had scored the case adequate:

[IMPORTANT] Narrative leans toward AI engineer/builder more than embedded business partner

Dimension: argument · found by 2 of 18 review instances

— Opus 4.6 / anchored-mechanism, summary: "the resume reads more like an AI engineer application than [an application for the role]."

— Opus 4.6 / anchored-combined, summary: "misses the JD's core framing as an embedded business partner and coach, instead positioning the candidate as an AI engineer/builder."

— Opus 4.6 / anchored-combined, overall.evidence: "The argument is built around 'I build AI solutions' rather than 'I embed in teams, identify opportunities, build solutions, and leave teams more capable.'"

That is the human verdict, recovered verbatim from runs that scored argument at 1. Alongside it the report surfaced the thin coaching and change-management evidence and the unused adoption and ROI proof — the other two things the labels require for this case.

It also got the ranking wrong. It marked the missing Microsoft Copilot and Power Platform keywords as the single blocking finding and left the targeting failure at important. The human verdict is the reverse: the ATS gap is real but survivable, and the targeting failure is the fatal one. The report finds the right problem and files it second.

Then I ran the other fifteen

One case is a demonstration. The full set is a measurement, and it is a considerably harsher one. Sixteen cases, same configuration:

The mistargeted case is in the rejected column. It passed on the standalone run and failed on the batch run under identical settings — the model misquoted a single source the second time. So the demonstration above was a lucky draw, and report-level yield on this configuration is 44%.

That is not a model-capability problem. The synthesizer misquotes roughly one citation per report, and the validator is all-or-nothing at report scale: one bad quote out of twenty discards seven correct findings along with it. The granularity is wrong. Rejecting the finding rather than the report would have salvaged eight of the nine failures. The tooling also discards the raw response on rejection, so neither the good findings nor the offending quote survives for diagnosis — which is why the failing quotes are not reproduced here.

And the diagnosis flattens. The hollow-history case passed validation, so its report is intact. Its core finding — a three-year role represented by one sentence and no bullets, which the human bands at basics = 0 — does not appear. The evidence does. The first finding quotes it exactly:

"the resume reduces this entire three-year role to a single sentence about being a 'technical point person.'"

But it is filed under argument as "Charter experience is compressed enough to hide relevant AI and system-design work" — a content-coverage problem. The only basics finding in the report is [minor] One omitted role creates an apparent timeline gap. A structurally hollow work history has been re-diagnosed as a local omission.

That is precisely the transformation the synthesis contract forbids — "an omitted detail is not the same diagnosis as a whole-document targeting failure" — and it is the same shape as the mistargeted case's severity inversion. Across both core cases that produced a report, the collection step recovered the evidence and then flattened the whole-document failure into a local one. It inherits the blind spot it was built to correct.

The most interesting section was one I did not anticipate. Because the contract asks for contradictions rather than consensus, Gemini's inversion became visible instead of averaging away:

Overall targeting quality for [the role]

— Gemini 3.1 Pro, summary: "This is an exceptional resume that accurately and compellingly targets [the role]."

— Opus 4.6, summary: "misses the JD's core framing as an embedded business partner and coach, instead positioning the candidate as an AI engineer/builder."

A scoring pipeline would have averaged those into a shrug. The collection step puts them side by side and makes the disagreement the finding. That does not rescue Gemini — costing two false accusations to serve as a foil is a bad trade, and it should still be excluded — but it shows the architecture degrades gracefully when a judge is wrong, which the scoring channel does not.

The hard ceiling

Union is not magic. Six required findings were never produced by any of the twelve configurations: the imperative summary voice, inline keyword bolding, length-driven keyword spam and unfocused writing on the longest résumé, an irrelevant credential left in, and awkward phrasing. No collection recovers a finding no review contains.

The pattern is not random. These are perceptual — properties of emphasis, rhythm, and visual weight rather than content. The controlled pairs measure the same wall:

One effect replicated. Three further independent runs confirmed that the pressure passage separates the buried-lead pair for GPT-5.4 on all four runs, base on none, mechanism and combined on two of four each. Voice and bolding did not replicate, and the two later integrated prompts lost the salience effect entirely — both zero of six — while the harsh one also cost Opus the inflated-seniority catch it had held under all four earlier prompts. Consolidating two passages that each worked separately produced a prompt worse than either.

Voice shows what "perceptual" means. Under the two integrated prompts Opus finally dropped the defective twin's writing score by a point, citing "keyword threading" and bullets "overstuffed with buzzwords," quoting the injected imperative summary as its evidence. The right sentences, the wrong diagnosis. Not one of the twelve claim-checked reviews named the imperative voice.

The answer is not a better prompt. Inline bold spans can be counted. Summary sentence order can be diffed. These are deterministic checks mistakenly assigned to a language model, and they belong on the lint rail beside truncation.

What to expect on architecture variations

Do not rank topologies on evaluator scores. On functioning résumés overall is near-constant by construction. The live model sweep confirmed it independently: eight model combinations on one input returned an identical overall for all eight. That was recorded as a tie; it is better read as an instrument with no resolution.

Do compare on collected findings. Run each variation through several reviews — two models across a few prompts — and compare the findings reports: how many blocking and important findings each architecture's output draws, and which ones. That comparison is supported by the measurements, at 73% coverage of what a human would flag with no false accusations.

Do not run the collection step unattended yet. Report-level yield is 44%, and a rejected report is discarded whole. Until validation moves to per-finding granularity, every synthesis needs a human to notice it did not produce anything.

Budget for a real synthesizer. The cheap post-processing model fabricated fifteen citations. GPT-5.4 at medium effort with 16k of headroom costs about seven cents per case and still misquotes roughly one citation per report. Cheap, but not free, and not a mini-model job.

Do not trust severity or dimension. On both core cases the report recovered the evidence and then filed a whole-document failure as a local one. Read the findings and their quotes; ignore the ranking and the dimension label until that is fixed.

Gate on grounding and lint. Fabrications 18 of 18, truncation 18 of 18. Safe to automate today.

Exclude Gemini 3.1 Pro. Redundant on coverage, sole source of false accusations, inflating in every configuration.

Expect a blind spot in typography and order until the perceptual checks move to the linter.

What is actually missing

Not a replacement evaluator. The judges supply the material; the collection step loses it. Four fixes, in order of value per unit of work:

1. Validate per finding, not per report. Eight of nine rejections were a single bad quote. Dropping the offending finding and keeping the rest would take yield from 44% to something near complete, and costs almost nothing to implement.
2. Persist the raw response on rejection. Right now a failed synthesis destroys both the correct findings and the evidence needed to diagnose the failure.
3. Stop the diagnosis flattening. On both core cases the step quoted the right evidence and downgraded a whole-document failure to a local omission — the exact move its prompt forbids. The labels already mark which findings are core; the contract does not consume that, and a separate whole-document targeting pass may be the more honest structure than asking one prompt to resist the flattening.
4. Perceptual defects to deterministic checks. Bolding, salience, and voice are countable properties of the document. They have absorbed more prompt engineering than anything else here and produced one replicated effect.

Verdict

The experiment did what an instrument should: it falsified the thing it was built to certify, then showed the failure was in the wiring rather than the component. There is no configuration here I would use to score and rank generation pipelines, and my earlier recommendation to gate on GPT-5.4 with the pressure passage overstated one narrow pairwise result.

What is real is more useful than what I set out to prove. These judges read résumés considerably better than they grade them — including the case they grade most wrongly, where the correct diagnosis was recoverable, verbatim, from the same runs that got the number wrong. The union of eight reviews contains 73% of what a human would flag, with zero false accusations and all three core failures present. That material is genuinely there.

The collection step does not yet get it out. It rejects more than half its reports over single misquotes, and on both core cases it recovered the evidence and then flattened a whole-document failure into a local one — reproducing downstream the exact blind spot it exists to correct. That is a sharper and more tractable problem than the one I started with. The reading is good, the collection is where the loss is, and every fix on the list is plumbing rather than research.

Data and instruments live in the resume-lambda repository: cases and labels under tests/eval_calibration/; matrices, grading reports, and claim-check verdicts under test_results/evaluator-batches/ in anchored-matrix-16case-3judges-20260810, anchored-mixed-negging-16case-3judges-20260811, percept-repeats-gpt54-*, and claim-check-matrix-20260810; the synthesis runs under test_results/finding-synthesis/; the collection step in src/versioning/finding_synthesis.py.