Résumé Generator

Check it out here: Resume Generator

Introduction

This project is an extension of a substantial amount of work done to automate one of life's occasional necessities, drafting resumes. Rather than spam a stock resume or embrace the time consumption of drafting tailored resumes, the modern era of writing machines enables a third way: LLM resume generation.

The basic idea is simple: provide the LLM with a document describing the candidate's background, the job description, and instructions for writing the resume, and let it follow its bliss.

It began as an exercise in wiring an LLM into a Lambda, but it evolved into an assessment of different generation graphs, which in turn pushed the focus toward evaluation: writing the judges and building the harnesses that let one architecture be compared honestly against another.

This is not without risks, however. Problems observed include the following:

• Inaccuracy: ungrounded or "hallucinated" statements, such as combining jobs
• Mis-structured outputs: including code fences, botching HTML
• Cost: needing complex instructions in turn requires large models to follow them well
• LLM writing dysfunctions: spamming technical buzzwords and some punctuation marks — like the em-dash

Initial development

Initially it was apparent that what is needed is a general discussion of the user's work history, education, and so on. This can be fed to the LLM along with the job description, and a résumé comes out the other end.

If you're into markdown. And nowadays, are we not all down with markdown? And yet, it's not the standard, "approved", proprietary compressed XML format systems expect; nor the readable, immutable PDF; nor even plain text. Instead it uses readable-ish punctuation sigils and hopes the user will put it through a formatter.

No, plain markdown would not do. PDF is the right choice: it always displays correctly and provides peace of mind and a feeling of professionalism. So the first task was creating a system that will turn something an LLM is comfortable producing into a PDF. We will skip recalling the attempt at for everyone's comfort. The answer converged upon was to use the Chromium engine and custom CSS alongside a charming Markdown/HTML hybrid ultimately christened "Power Markdown". Much futzing with CSS and document formatting ensued.

Given candidate's longform history, the job description, and Power Markdown examples, all that's left is feeding the beast and telling it to write a resume. Simple. And if the assistant can search around in the local directories and find fine-tuned examples, boosting its own output, so much the better. Everything seems fine.

Operationalization

For creating the Résumé Generator in the site, more is required. Finding Chromium-sporting images for the PDF renderer, wiring up Cloudflare Turnstile and the API gateway, doing AI Native fullstack development to make the JavaScript component for the frontend, job description processing, storage and metadata setup, and of course, the main serverless instance which interfaces with these to call the LLM API.

Crafting the LLM request (for Claude) was not especially difficult. The prompts used locally had to be split out into system and user prompts, and a preprocessor was created to build the user prompt using the longform history (a versioned and stored input) and job description.

Issues and Opportunities

The main issue discovered was that LLMs really like technical keywords. Why should a 4-trillion parameter Transformer decoder trained on the sum total of human knowledge (i.e. Reddit) fight its natural affinity with ATS systems? Prompt-massaging only goes so far in this circumstance, somewhat surprisingly.

The solution was to split the generation flow into two stages, both with strong instructions steering the models toward human-oriented, meaning-first output. The first phase acts as a résumé strategist: it reads the job description and the candidate's history and returns a brief, deciding what to emphasize and which evidence supports it. The second phase is a composer that writes the résumé from that brief, so the model doing the prose is never left to free-associate over raw inputs.

That split raised an immediate question, though: how do you know a given pipeline, or a given model, actually produces a better résumé than another? Which is where the evaluator comes in.

Evaluating the Evaluator

Comparing graphs and models sounds straightforward until you ask the obvious question: compared how, and judged by whom? The judge, inevitably, is another LLM with its own tastes and blind spots. So before trusting any leaderboard of prompts and pipelines, we had to turn the lens around and evaluate the evaluator.

The tool for that is a calibration set: a small, deliberately varied collection of résumé and job-description pairs, each one a probe for a specific judgment. A couple are clean references, the picture of what "good" should look like. One is an outright failure the deterministic linter is meant to catch, a résumé that trails off with no work history at all. One pairs a strong candidate with a role beneath them, to check that the judge separates "is this a good résumé?" from "is this person a fit?", which are emphatically not the same question. And the revealing ones are deliberately salted with flaws no linter can catch, each one planted to test a single judgment. A couple were hand-edited into otherwise clean outputs; the rest were kept from an earlier, blunter prototype, precisely because it made the mistakes the current pipeline is built to avoid:

• a summary rewritten into a bossy, job-posting register ("Design and deploy…") in place of the candidate's own voice, to test the judge's ear
• a planted, unsupported skills section, to test whether it catches an ungrounded claim
• an off-target credential left in against the brief, to test whether it flags the irrelevant
• inline keyword spam

These are decoys, in other words: known-bad specimens whose only job is to reveal what a given judge does and does not notice. Each runs through the evaluator while we swap the judge across the three big model families and climb each provider's full reasoning-effort ladder, up to max for Claude and Gemini and up to xhigh for GPT-5.4 (OpenAI rejects max outright).

The anchors behaved, which was the first relief: truncation floored to zero, clean résumés scored high, and the down-leveled candidate kept a high mark, fit and quality held properly apart. The harness measures what it claims to.

The sharp result concerns that bossy-voice summary. Whether a judge notices it turns out to be a trait of the model, not of how hard it thinks, and climbing the whole effort ladder made that vivid. Here is the writing score on the voice case at each rung, where 3 means the defect was caught and 4 means it sailed straight through:

(Top rung is max for the Opus models and xhigh for the GPT models. Sonnet 4.6, run only at high and max, also catches it; all three Gemini judges, on adaptive thinking, miss it outright.)

The result is stranger than "thinking doesn't help." For two of the judges, thinking actively hurts: Opus 4.7 sees the defect at low and goes blind the instant it deliberates harder, and GPT-5.4-mini holds on all the way up before dropping it at the very top. Only Opus 4.6 and full GPT-5.4 stay right the whole climb; Opus 4.8 and every Gemini stay wrong. So more reasoning can make a judge more forgiving, not less, and none of it teaches a blind judge to see.

Thinking does earn its keep in one narrower way: it sharpens the judges that already see, trimming the lazy ties in the middle of the scale. Full GPT-5.4's flat 0.75 verdicts thin out as it thinks harder:

The lesson is unglamorous and useful: pick a judge for whether it can actually see the failure you care about, and treat reasoning effort as a dial that can sharpen a clear judge or blunt a shaky one, but never rescue a blind one.

The rest of the board

The voice case is the extreme, but the other decoys sort the judges into a rough gradient of perceptiveness. The one flaw most judges catch is the fabricated skills section, the kind of ungrounded claim the whole pipeline exists to prevent. An off-target credential slips past far more often, and keyword stuffing and the bossy voice sail through almost untouched (counts are across all twenty-five judge runs):

So the judges are decent grounding cops and poor stylists: ungrounded content draws at least some scrutiny, ungrounded tone draws none. But no judge is a clear champion, and that turns out to be the real finding. The single flag on the keyword spam came from Opus 4.6 at medium effort, and from nobody else at any setting. The off-target credential was caught by a scattered four (a couple of Opus 4.7 runs, GPT-5.4 at low). The steadiest hand is full GPT-5.4, which dings the fabricated skills at every effort level, yet even it misses the subtler cases like everyone else. When the "sharpest" judge changes depending on the rung, you are looking less at skill than at scatter.

The control cases, meanwhile, behaved as hoped, which is what earns the rest of the numbers their credibility:

One methodological wrinkle is worth stating plainly. The two PDF-based dimensions, layout and ats, are close to noise. On the clean reference résumé, the one with nothing wrong with it, layout scores ranged across the entire scale, from 1 to 4, depending only on which model happened to be looking. A single judge's layout verdict is barely better than a coin flip, so those dimensions are best read as mood, not measurement, and the interactive table below lets you see that scatter for yourself.

And how much of any of this survives a second look? Less than the decimals imply. The same judge, at the same effort, run a second time by a different operator on identical inputs, agrees with itself on the overall score for as few as five of the ten cases, with the average gap running about half a rubric point. The catch-or-miss verdicts are stable, which is why they carry the argument; the fine gradations drift from run to run. It is why the honest unit here is "did it see the defect," not "0.75 versus 1.0," and why an apparent standout judge dissolves the moment you add more runs.

There was also a comeuppance for enthusiasm. One thinking model kept truncating its own verdicts: handed an explicit budget for deliberation, it would ponder right up to the edge of its output allowance and then run out of room mid-object, leaving a beautifully reasoned, syntactically broken half-answer. The fix was to reserve headroom for the answer on top of the budget for the thinking. A small thing, but a tidy reminder that thinking and speaking draw from the same well. With the fix in place, the full ladder ran clean: across all twenty-five judge runs, not one truncated verdict.

Appendix: the full data

Every judge on the full effort ladder, every case, every dimension, across the ten-résumé calibration set: twenty-five model-and-effort combinations in all (the two OpenAI max runs are omitted, since the API rejects that setting). Scores are the evaluator's raw rubric judgment, from 0 (fail) to 4 (excellent). Pick a dimension and toggle judges to explore.

Comparing generators

With a judge we can at least partly trust, the point of all this scaffolding comes into view: comparing generators head to head. Two sweeps are in flight.

The first is a model sweep over the two-stage pipeline. Holding the architecture fixed, it varies the strategist and composer models (a small, fast model for strategy against a larger one; several choices for the composer) to find where quality per dollar actually lives.

The second is an architecture sweep. Holding the models roughly fixed, it compares topologies: a single prompt, a generator-with-critic loop, a cyclic revise-until-good chain, and a parallel fan-out of evidence collectors and critics, to see whether the more elaborate graphs earn their extra calls.

Both are scored on the same calibration-checked evaluator described above, with cost and latency tracked alongside quality, since a graph that spends a dozen model calls to win by a hair is not obviously a win.

Model sweep smoke results

The first live pass was deliberately small: one shared job-description input, eight model combinations, local HTML/PDF artifacts, and the post-hoc evaluator attached to every successful generation. Before running it, the harness needed two pieces of plumbing fixed.

First, evaluator requests now carry the same experiment identifiers as the generation requests: experiment, variation, input label, generation graph, and generation trace IDs. That matters because otherwise the evaluation span can look detached from the exact generation it judged. Second, GPT-5-family calls no longer receive temperature; the provider rejects it for these reasoning-style modes. The GPT composer variants also needed more output headroom: medium reasoning worked at 8192 completion tokens, while high reasoning returned empty content until raised to 16384.

On this first input, the evaluator gave every successful generator the same overall score, so the early signal is not "winner by score" so much as "which settings are viable, cheap, and worth human review." The table below uses the locally recorded provider cost where available; the evaluator cost was not fully priced in the local map, so the dollar column is best read as a lower-bound run cost.

The main result is practical rather than dramatic. The cheaper two-stage baseline remains competitive on this case, and larger strategy/composer models did not buy an obvious evaluator-score jump. The next pass should therefore compare the saved PDFs and markdown by hand before expanding the matrix across more inputs. If the human review also shows a tie, the smaller strategy model plus Sonnet composer is the sensible default until a harder case proves otherwise.

The architecture sweep remains the larger open question. The model sweep says a two-stage graph is viable and relatively cheap; it does not yet answer whether critic loops or parallel evidence collection earn their extra calls.