Résumé Generator

(draft, large sections of generated content)

Check it out here: Resume Generator

Table of Contents

1. Introduction
2. Architecture and Deployment
3. Pipeline Enhancements: LangGraph, Langfuse, and SQS
4. Creating the Evaluator
   1. Evaluation Playground — twin-pair comparisons and the roll-up explorer
   2. Detailed evaluation reports
5. Generation Graphs
   1. Generation Flows and Deterministic Controls
6. The Résumé Validator
   1. Validator design and findings
7. Conclusion

Introduction

This project is an easy win built off work done to automate one of life's occasional necessities, drafting resumes. Rather than spam a stock resume or pay the time cost of drafting tailored resumes, the modern era of literate machines enables a third way: LLM resume generation.

The basic idea is simple: provide the LLM with a document describing the candidate's background, the job description, and instructions for writing the resume, and let it follow its bliss. Of course it's more complicated than that.

What began as an exercise in wiring an LLM into a Lambda (the system lives in a repository called resume-lambda) grew in two directions at once: upward into a comparison of different generation graphs, and downward into evaluation, because comparing graphs honestly turns out to require knowing whether your judge can actually judge. Much of the interesting work, and most of this page, concerns that second part.

Architecture and Deployment

Initial development focused on the core workflow, using a single-prompt approach hosted by coding assistant apps, which facilitated rapid iteration on basic elements of the system. The goal was a PDF file, and the format that emerged was a mixture of HTML and Markdown, colloquially called "Power Markdown", which is fed through a headless Chromium renderer along with custom CSS to produce the desired formatting. Much futzing with CSS and document structure ensued, but the result reliably turns something an LLM is comfortable writing into a document a human is comfortable sending.

The move from apps to direct LLM API requests represented the author's first use of these services at the developer level. Most impactful was the power of controlling the system prompt, which drastically improves the compliance of the LLMs in output creation. A preprocessor builds the user prompt from the candidate's longform history (a versioned, stored input) and the job description.

Operationalizing this for the site required the usual pile of supporting engineering: finding Chromium-sporting images for the PDF renderer, wiring up Cloudflare Turnstile and the API Gateway, AI-native fullstack development of the frontend JavaScript component, job description processing, S3 storage and metadata setup, and of course the main serverless function that interfaces with all of these to call the LLM API. Deployment is handled with AWS SAM.

An early and necessary test of the deployed service was robustness against malicious job descriptions, which are ultimately included in the user prompt and could serve as a vector for prompt injection. While the actual risk is low, given the system is designed for public output, a handful of red-team job descriptions were made: both "jailbreak" attempts to get the LLM to emit its system prompt, and job descriptions for unsavory roles that one wouldn't want to produce a resume for, due to reputational risk. Interestingly, both approaches triggered refusals by the source systems to continue in execution, demonstrating a basic level of safety and taking some burden off the developer.

Pipeline Enhancements: LangGraph, Langfuse, and SQS

In its first deployed form the Lambda performed job description processing and resource setup (configuring the S3 directory for the artifacts and whatnot), then would self-invoke to kick off the API calls. Self-invoke is a "meh" pattern, and it would have become even more "meh" with multiple processing steps, so the system was migrated to use SQS pipes to trigger the different Lambdas, resulting in a cleaner architecture.

As the generation flows grew stages, orchestration moved to LangGraph, which turns each flow into an explicit graph with typed state passed between nodes. This is what makes the graph families discussed below practical to register, version, and compare against each other; the registry currently runs from main/1 through main/19.

Observability came via Langfuse. Model calls are traced, prompts are versioned, and generation and evaluation runs carry shared identifiers (experiment, variation, input label, generation graph, and trace IDs), so an evaluator's verdict can always be tied back to the exact generation it judged. This sounds like bookkeeping, and it is, but it started paying for itself the moment the experiments below got serious.

Creating the Evaluator

The multi-stage flows raised an immediate question: how do you know a given pipeline, or a given model, actually produces a better resume than another? The judge is inevitably another LLM with its own tastes and blind spots, so before trusting any leaderboard of prompts and pipelines, the lens had to be turned around and the evaluator itself evaluated.

The evaluator is two independent rails. A deterministic linter checks truncation, structure, and the candidate's employment titles and dates, with no model judgment involved. A qualitative review then scores six dimensions (basics, writing, argument, grounding, keywords, overall), each returning a judgment, a rationale, and evidence.

The first calibration round ran ten resumes salted with known defects — a fabricated skills section, keyword spam, an off-target credential, a summary rewritten into bossy job-posting voice — across twenty-five judge and reasoning-effort combinations from the three big model families. The topline was that the judges are decent grounding cops and poor stylists: ungrounded content drew at least some scrutiny, ungrounded tone drew essentially none, and whether a judge noticed a given defect was a trait of the model rather than of how hard it thought. More reasoning effort sharpened judges that could already see, and blinded at least one that could: Opus 4.7 caught the voice defect at low effort and lost it as soon as it deliberated harder. Repeat runs also showed the fine score gradations drifting while the catch-or-miss verdicts held, so the honest unit of measurement is "did it see the defect", not the second decimal place.

The second round rebuilt the instrument properly: a unified 0–2 rubric, sixteen cases with human-ratified score bands and required findings, five defect-injected twin pairs, a controlled 2×2 experiment on reading-instruction prompt passages, and a small model cross-examining every review for what it actually said. The results were uncomfortable but clarifying. The scoring channel is close to inert: on the hardest case, a polished engineering resume mis-targeted at a hybrid, business-facing role, the eighteen judge configurations produced zero variance — each judge returned the same score under every prompt, and none of them reached the human's verdict. The reading channel is live: the same instruction passages that leave the scores flat lift what the review prose correctly mentions by fifteen to twenty points, without purchasing that recall with false accusations. The numbers cannot rank pipelines, but the prose contains most of what a human reviewer would flag; the union of two judges' reviews covered 73% of the required findings.

That pushed the work toward collection: a second-stage model reads the evaluator reviews and compiles the prose into findings, each citing a verbatim quote that code re-checks against the source review. A sweep of second-stage models found Gemini 3.5 Flash-Lite to be a reliable, inexpensive reader — 230 of 240 reviews compiled into valid source-cited reports for about 46 cents, versus 183 for $2.66 from Haiku 4.5 — and a follow-up label analysis confirmed the roll-ups mostly compress rather than distort. The remaining failures are wiring rather than models: validation is all-or-nothing per report, and whole-document diagnoses have a tendency to get flattened into local ones on the way through.

That is where the evaluator work presently stands: a lot of data, with the conclusions still being drawn — by hand, in the evaluation playground, which puts the twin-pair comparisons and every roll-up from the full 240-review matrix side by side for inspection.

Detailed evaluation reports

The detailed writeups, in rough chronological order:

• The Evaluator Reads Well and Scores Badly — the sixteen-case anchored matrix: 288 judgments, the targeting failure no configuration caught, and the corrected verdict on the collection step.
• Mixed-Prompt Qualitative Rollup — the collection stage over all sixteen cases from the GPT-5.4 and Opus 4.6 reviews, with per-case rollup pages.
• Single-Review Synthesis Pilot: Flash, Flash-Lite, and Luna — a small pilot of the single-review collection stage across five prompt treatments.
• Full-Matrix Finding Synthesis: Haiku vs Flash-Lite — both models over all 240 source reviews, with measured yield and cost.
• Do the Evaluator Roll-ups Match the Human Labels? — what the evaluator prose actually says versus what a human labeled, plus a manually adjudicated section on the hardest case.

Generation Graphs

Three families of generation graph have been implemented, in increasing order of decomposition:

• V1, unified. One model reads the raw inputs and writes the finished resume; code strips provider scaffolding and lints the result. Simple, and the baseline everything else has to beat.
• V2, two-stage. A strategist reads the job description and the candidate's history and produces a brief: what to emphasize and which evidence supports it. A composer then writes the resume from that brief, so the model doing the prose is never left to free-associate over raw inputs. This split was the original fix for the keyword-spam problem, which prompt-massaging alone somewhat surprisingly failed to solve. The V2 beta experiments wrap critics, revision loops, and parallel specialist passes around composition.
• V3, fan-out. The job description is decomposed into independently searchable desired attributes, an evidence miner runs per attribute, a winnowing step selects a global evidence plan, and composition happens last. The structured variants go further: evidence excerpts are validated verbatim against the source record, selection happens by immutable ID, and code — not the model — assembles the role containers and renders the final document.

The graph-by-graph tour, including flow diagrams and exactly which decisions belong to models and which are enforced in code, is on the generation flows page.

Head-to-head evaluation of the graphs is not ready yet, and the evaluator section above explains why. A smoke-test model sweep (eight strategist/composer combinations on a single input) ran end to end and mostly proved out the plumbing: evaluator spans now share identifiers with the generations they judge, GPT-5-family calls needed temperature removed, and high reasoning effort needed more output headroom before it would return anything at all. The evaluator gave every successful combination an identical overall score, which is less a tie than an instrument with no resolution. The graph comparison will resume once the instruments deserve it.

The Résumé Validator

Whatever else a resume gets wrong, grounding is the question that has to be answered correctly: does the document say anything the candidate's record doesn't support? A resume that reads beautifully and fabricates is worse than useless. Conveniently, this also turned out to be the question that could be shrunk until models answer it well.

A first validator tried to make grounding checkable by structuring it: extracting subject–predicate assertion tuples from both documents and comparing them with deterministic rules. It failed quietly, because all the subjectivity got pushed into the extraction step, where honest sentences shattered into partial tuples and near-verbatim copies of the source read as overstatements.

The rebuilt validator follows the decompose-then-verify pattern from the fact-verification literature. Stage one restates each resume line as atomic claims in plain language, each carrying a verbatim quote from the line that code verifies character-for-character. Stage two judges each claim alone against the entire candidate record, with three verdicts that are not allowed to blur: supported, contradicted, or unsupported — and supported or contradicted verdicts must quote the record, also checked in code. The judgment stays with the model; the lying gets hard.

The control set was built from defects that occurred naturally in generator output: a bullet inflating product ownership into "led end-to-end development", a web portal promoted to a "SaaS portal", a component attached to the wrong parent system, in-progress work claimed as shipped — each paired with a clean twin line to catch false alarms. Cheap runs with Haiku on both stages scored 7/9 and 6/9, with misses that were diagnostic gold (decomposition, not judgment, was the bottleneck). With GPT-5.6 Terra on both stages the validator went nine for nine: 511 claims decomposed with zero quote violations, every defect caught with a verbatim receipt, every clean twin left alone.

This part of the program, cautiously, appears to work. It is an instrument check rather than a field trial, but nine for nine with receipts is the first result in the whole evaluation effort that behaved the way one hoped. The design, the failure analysis of the cheap runs, and the full scorecard are on the claim verification page.

Conclusion

The system began as a prompt in a Lambda and became a pipeline with an explicit division of labor: models make judgment calls and write prose, while code owns identity, attribution, ordering, formatting, and increasingly, verification. Every move in that direction has been rewarded.

The evaluation work delivered a conclusion too, just not the one it was commissioned for. LLM judges read resumes considerably better than they grade them: holistic scores are flat to the point of uselessness, while the prose from the same runs contains most of what a human reviewer would flag. The pattern that survived contact with the data — shrink the question, fence every answer with a deterministic check — is the backbone of the validator, and the standard against which the rest of the evaluation stack is being rebuilt.

Next up: running the validator over the organic calibration cases where the holistic judges failed, moving roll-up validation to per-finding granularity, and then returning to the question that started all of this — whether the fancier generation graphs actually earn their extra calls.