Evaluation playground

This page is a working surface, not a writeup. The detailed analyses have their own pages; the goal here is to put the interesting slices of the evaluator data side by side so that conclusions can be drawn by an actual human. Everything below comes from the anchored sixteen-case calibration matrix: three judges (GPT-5.4 at high effort, Opus 4.6 at adaptive medium, Gemini 3.1 Pro at high), five prompt treatments, and the two second-stage roll-up models (Haiku 4.5 and Gemini 3.5 Flash-Lite) run over all 240 individual reviews.

The cases

Every resume in the set is a real generation for the same candidate against a real job posting; the companies and posting titles are withheld here. Five cases stand alone, five come as clean/defect pairs, and the last (Truncated) is the floor check.

The five pure cases

These are unmodified pipeline outputs, each carrying my review notes as the ground truth the judges are graded against.

• Mistargeted — a polished, keyword-heavy resume aimed at an embedded, business-facing AI role. The posting wants initiative road-mapping, cross-functional collaboration, and user coaching; those connections are reachable from the candidate history, and the resume never makes them, arguing "engineer" instead. Demoted from a positive anchor after my review, it now serves as the leniency probe: a judge that scores it high on overall, writing, or argument has missed the point of the document.
• Skills-spam trap — an older-pipeline output for a product role, originally labeled as fabricated skills spam. My review inverted that: the skills-spam verdict is not well supported, and flagging it is a false positive. The real defect is that the posting's UX-research responsibilities go unaddressed, even though they are inferable from the customer-facing portal work and an advanced degree in economics.
• Credential trap — an older-pipeline output for a process-improvement analyst role. The off-topic graduate credential it includes is present in the pinned candidate snapshot, so calling it a grounding failure is a false positive. The real defects: a pile of cloud and architecture bullets that are off-target for an analyst role, flat writing, and connections to the job's needs that are never argued.
• Long and spammy — the longest resume in the corpus at roughly 1,130 words. Flipped by my review from "long but clean": the length produces keyword spam and an unfocused feel, and that is what should be penalized, not the word count itself.
• Hollow history — a complete-looking resume in which a three-year role is reduced to a single sentence with no bullets. Nothing is cut off; the substance was simply never written. The posting is research-heavy, so fit is a stretch on top of it.

The five manipulated pairs

Each pair is one resume in two versions differing by exactly one manipulation, so a judge's clean-versus-defect delta isolates whether it perceived that one thing.

• Voice — clean: the original output, its summary written normally. Defect: the summary hand-rewritten into reader-addressed imperative ("Design and deploy… Translate… Drive…"). My notes: the voice issue is clear; besides that, the resume is actually pretty good.
• Bolding — clean: the seventeen mid-sentence inline keyword bolds unwrapped, keeping the fourteen house-style bullet lead-in bolds. Defect: the inline bolding left intact. My notes: this is a highly technical JD, so the keyword emphasis is less out of place than it could be; the real absolute issues are bullets that lack consequence and under-claimed collaboration.
• Pitch — clean: honest solo-developer/analyst framing. My notes: wordy, with the posting's adaptability points under-addressed; the old "down-level" premise is withdrawn, and asserting it is a false positive. Defect: three seniority inflations with the facts otherwise intact — "builds … as a solo developer" becomes "leads production AI application development", "supported … delivery" becomes "directed", and "worked with cloud operations and security teams" becomes "led" them.
• Salience — clean: the human-validated high anchor; this exact resume got a phone screen, and my notes call it very good — automated reviewers that penalize its length are wrong to. Defect: a pure permutation — summary sentences swapped so the AI-shipping clause is buried, and bullets reordered weakest-first within each role. Identical words throughout; a reader that only queries content cannot distinguish the pair.
• Grounding — clean: a lean resume against a sparse, 278-word posting. My notes: the candidate clears the stated requirements with room to spare, so treating the sparsity as fatal is a false positive; OK to good, not great. Defect: three injected fabrications — a blatant fake cloud certification, a 6% performance lift inflated to 16%, and a 15× speedup inflated to 40×. Full credit means catching a subtle inflation, not just the fake credential.

The sixteenth grid case, Truncated (floor check), is an organic generation failure that cuts off partway through the professional summary — broken and unusable, though what's there reads fine. It exists to confirm that hard failures floor the score.

The twin pairs

The five pairs described above, compared side by side. Because each pair differs by exactly one manipulation, the clean-versus-defect delta isolates whether a judge perceived that one thing. Scores are shown on the human 0–2 rubric scale.

The roll-ups

The second stage reads a single evaluator review and compiles it into source-cited findings. Below is every roll-up from the full 240-review matrix. A deterministic verifier checks each citation verbatim; roll-ups that failed that check (usually over a single bad quote) are shown anyway, recovered from the raw model output and marked unvalidated. One Haiku call out of the 480 returned no text at all, and that cell stays empty. Pick the synthesizer, get the overview of what it produced for every case, then drill into any single roll-up.

.source-review,
.output-card {
  border: 1px solid color-mix(in srgb, currentColor 18%, transparent);
  border-radius: 0.7rem;
  padding: 1rem 1.1rem;
  background: color-mix(in srgb, var(--theme-background, white) 97%, currentColor);
}
.source-review > header,
.output-card > header {
  display: flex;
  justify-content: space-between;
  gap: 1rem;
  align-items: baseline;
  flex-wrap: wrap;
}
.source-review section {
  border-top: 1px solid color-mix(in srgb, currentColor 12%, transparent);
  padding-top: 0.5rem;
}
.source-review h4,
.finding h4,
.output-card h3 {
  margin: 0.5rem 0 0.3rem;
}
.output-grid {
  display: grid;
  grid-template-columns: repeat(auto-fit, minmax(min(100%, 28rem), 1fr));
  gap: 1rem;
  align-items: start;
}
.output-card.invalid { border-color: #b91c1c; }
.output-card.valid { border-color: #15803d; }
.output-card.recovered { border-color: #b45309; }
.status,
.severity {
  border-radius: 999px;
  padding: 0.15rem 0.5rem;
  font-size: 0.73rem;
  font-weight: 700;
  letter-spacing: 0.035em;
}
.output-card.invalid .status { background: #b91c1c; color: white; }
.output-card.valid .status { background: #15803d; color: white; }
.output-card.recovered .status { background: #b45309; color: white; }
.severity-blocking { background: #b91c1c; color: white; }
.severity-important { background: #b45309; color: white; }
.severity-minor { background: #475569; color: white; }
.finding {
  border-top: 1px solid color-mix(in srgb, currentColor 14%, transparent);
  margin-top: 0.9rem;
  padding-top: 0.8rem;
}
.finding blockquote { margin: 0.7rem 0; }
.raw-output {
  max-height: 34rem;
  overflow: auto;
  white-space: pre-wrap;
}
.disagreements > h4 {
  margin: 1.1rem 0 0;
  font-size: 0.78rem;
  font-weight: 700;
  letter-spacing: 0.06em;
  text-transform: uppercase;
  opacity: 0.75;
}
.plot-figure {
  margin: 1rem 0;
  max-width: 100%;
}
.scroll-x {
  overflow-x: auto;
}
.grid-caption {
  font-size: 0.82rem;
  opacity: 0.72;
  margin: 0.35rem 0 0;
}
.drill-source {
  margin-top: 1rem;
}
.drill-source > summary {
  cursor: pointer;
  font-weight: 600;
}