The whole machine, and what it would take to run it.

The system is split by failure mode rather than by feature. Job boards rate-limit, change their APIs and go down; language models time out and refuse; a browser driving a form is slow and stateful. Each of those is a different kind of unreliability, and putting them in one process means any of them can stop all of it. Split, a board outage cannot stop you applying to the jobs you already have, and a model that is refusing today does not stop the sweep collecting tomorrow’s.

Written in Java 21 on Spring Boot for the three pipeline services and the gateway, Python 3.11 on FastAPI for the browser service, React 19 and TypeScript for the front end, and Chrome MV3 for the extension. Schema changes go through Flyway — seventeen migrations, applied by one owner.

The stages are separate processes talking through the database rather than a chain of synchronous calls, and that is what makes the whole thing survivable. Fetch does not wait for enrichment; it fires a notification and moves to the next role. Enrichment runs alongside the sweep rather than after it. Nothing in the pipeline holds a result in memory that it has not already written down.

Restarting never starts a fresh sweep. New jobs are a deliberate act — a button, or a schedule. A system that re-swept on every restart would be indistinguishable from one that was never paced at all.

Jobs come from the systems employers actually post into — Greenhouse, Lever, Workday, Ashby, SmartRecruiters, Oracle Cloud, iCIMS — plus Adzuna as an aggregator, and company career pages where a company runs no API worth calling. Each is a public listing endpoint read one company board at a time. Where a career page is the only route, it is classified first — which system is this page actually running? — and only then read, because knowing the system tells you where on the page the jobs are.

A sweep is planned rather than run flat out. Every target role is paired with the boards that role actually appears on, and the pairs are worked through a fixed thread pool with a cap on how many run for any single role, so one role with a long board list cannot starve the others. The cost of a run is linear in the number of roles, which is exactly why the role list is a list you edit rather than a box you type anything into.

• A normal run takes the last twenty-four hours. A first run, or one that finds a gap in the record, widens to seven days and backfills it.
• Postings with no date are aged from when they were first seen, not assumed fresh. A board that omits a date is common; a pipeline that treats every such posting as new is a pipeline that applies to the same job every day.
• Deduplication is a fingerprint of company and role, not the URL. Several boards mint a fresh tracking link per session, so the same requisition arrives at a new address every sweep.
• Location is deliberately left out of that fingerprint. One role posted in four cities is one job.

Pacing is not politeness for its own sake. Swept flat out, at least one of these systems blocks the address at its CDN, and the run of refusals that follows looks exactly like every company delisting at once rather than like a block. That failure was diagnosed the hard way and the pacing exists because of it.

Some sources publish no API and return an empty shell to anything that is not a real browser with a real session. Those are handled by a separate service that runs an actual Chrome on a virtual display inside its own container, with a browser profile stored in a named volume so a sign-in survives a restart. It is a different failure mode from an HTTP call — slower, stateful, and worth isolating behind its own process for exactly that reason.

The ordering matters more than the mechanism. The API path runs first and answers what it can; the browser is spent only on what is left. Extraction is a two-tier funnel for the same reason — a structured endpoint is tried before a page is ever rendered, because rendering a page to read a paragraph that was available as JSON is minutes of machine time bought for nothing.

This is also the part that is deliberately capped. A browser reading pages at volume stops looking like a person, and the cost of that is the account doing the reading — so the daily budget is small and the pacing is not configurable upward from the dashboard.

A job in the corpus is a title, a company and a link. That is enough to store and to deduplicate and nowhere near enough to rank: the same title at two companies can differ by five years of expected experience and an entire technology stack. Enrichment claims one job at a time, fetches the description behind it, and asks a model to return the skills genuinely required, the experience band, the location, and the requirements the title never states.

• The model call uses your API key, not ours. Nothing about the pipeline needs a key of ours, and no description or résumé passes through a service we operate on your behalf.
• Keys are held as a pool. Requests spread round-robin, a rejected key is marked dead, a rate-limited one sits out a cooldown and rejoins — so one expired key is a slower run rather than an outage.
• Claiming one job at a time is what makes the stage resumable. An interrupted run leaves the rest of the queue exactly as it was, and the next start continues with whatever is still empty.
• A malformed response is a failure that is retried, never a partial record written as though it were complete.

One newline once broke this. A raw line break inside a model’s JSON string made every enrichment, every résumé profile and every match fail to parse — and because the failure was silent, everyone on the deployment was ranked against the same fallback profile. Structured output is now parsed strictly and the failure is loud.

Ranking is not a similarity score with a threshold on it. It is a sort on four keys in a fixed priority, and the order is the contract — there are regression tests whose only job is to fail if it changes.

Three checks run before the sort and remove candidates outright: experience beyond the allowance, a title the model identified as excluded, and a job that mentions none of your leading core skills. Demotion would not be enough — a fifty-job dispatch still reaches rank forty, so anything merely pushed down is still applied to.

• The corpus is shared across the deployment; the ranking is not. Every account ranks the same pool against its own résumé, its own roles and its own track.
• Your résumé is profiled for what kind of engineer it describes — read for intent, from the summary, projects and depth of skills, not just from job titles, because employment history alone answers the question correctly and uselessly.
• Declared facts beat the model’s reading. Your stated years of experience is authoritative over anything inferred from the document.
• Scoring is parallel and capped at a measured number of threads. Past that the limit is memory bandwidth rather than CPU, and more threads made it slower.

When it is on, a worker notices jobs queued to apply and hands each one to a coding agent as a self-describing work packet: your master record, the job description, the measured skill gap, the required output shape, and the rules that will be enforced on the result. What comes back is checked before it can be used.

• Every skill on a tailored résumé must appear in your declared vocabulary. Every figure must appear in your verified quantities. Companies, dates and degrees are copied verbatim.
• Your bullet counts are a floor. A generated document with fewer bullets in a section than your record holds is rejected outright.
• The page is filled by measuring the rendered PDF and choosing the largest type that still lands on one page, rather than by asking a model to write to a length.
• Every failure path ends with the base résumé going out, exactly as if the feature did not exist.

The work surface is plain HTTP rather than a bespoke protocol, so anything that can make a request is a first-class client — a coding agent, a shell script, or something nobody has written yet.

Each applicant tracking system gets a purpose-built driver because each is genuinely different, and none of the differences are discoverable from the outside. Workday relabels its own steps, so sections are routed off its progress rail rather than its headings. Oracle keeps education and experience as a card ledger that has to match your profile entry for entry. Greenhouse ignores typed text in the city field and needs the suggestion picked. A generic form filler reaches about the third field on any real one.

• Nine hand-written drivers, plus a generic engine that navigates to the form, classifies the fields and fills what it can when the system is unrecognised.
• The résumé is attached as a real file drop onto the real input, because browsers refuse a value assigned to a file field — and rightly so.
• A question that cannot be answered from your profile is never filled with something plausible. The extension raises exactly the fields it is stuck on and waits.
• The queue claim is a lock. Two browsers signed into the same account is the one failure that presents as jobs vanishing rather than as an error.

The extension ships without the debugger permission. An earlier version used it to reach fields that resist ordinary automation; that permission is a review problem and a trust problem, and the capability was rebuilt without it rather than justified.

The straightforward way to build this is to run the browser on the server — a headless Chrome in a data centre, filling forms for every user out of one address, unattended. It is easier to operate and it is what most of this category does. It is not what happens here, and the reason is not philosophical.

• The systems that matter are the ones that check. Large applicant tracking systems and the edge networks in front of them treat an automated browser on a data-centre address as exactly what it is; the application is challenged, silently degraded, or accepted into nothing.
• A server-side browser has to hold your credentials to be signed in as you. This one never signs in on your behalf: the session is the one you established yourself, in your own browser, and it never leaves it.
• Anything a server-side browser cannot answer has to be guessed at or abandoned, because there is nobody there. Here there is somebody there, so the third option — ask — exists.

So the applying half runs as a Chrome extension, in a browser you are signed into, on one tab you can watch, at a pace a person could plausibly work at. There is nothing to detect, because nothing is pretending: it is your browser, your session and your connection, and the only unusual thing about it is who is doing the typing.

The moment it stops is the important one. A driver that reaches a field it cannot answer from your profile does not fill it with something plausible — an application submitted with an invented answer is worse than one that was never sent — so it stops the run, holds the tab open, and asks. Three things reliably cause that: a question the employer wrote themselves that nobody has answered before, a document that was never supplied, and a check that exists specifically to establish that a person is present. The third is not something to be defeated. It is answered by the person the check is asking about.

Where you are standing decides how the ask reaches you, and both surfaces carry the same request. Inside the browser profile the extension is loaded into, it is drawn over whatever tab you happen to be on — being on a different tab from the form is the ordinary case, not the exception. Outside it — another Chrome profile, an editor, another screen — a small window opens instead, deliberately without taking focus, and an answer given there reaches the waiting run through the relay it is already polling.

• The clock is three minutes, and the run polls on a two-to-three second beat, so pressing Done is felt almost immediately rather than at the next sweep.
• A timeout is not a failure. The driver performs one submit and inspects the real response, because what the employer says back is a fact and a job failed by our own clock is not.
• The widget checks that it actually rendered where you are. A page whose stylesheet suppresses it is escalated past — restated as important, and beyond that into the browser’s top layer, where no ancestor of the page can reach it — and a placement on a tab nobody is displaying does not count as having asked.
• What it shows you is sanitised at the point the ask is raised rather than in each driver, so an internal stack trace cannot reach the screen through a portal that forgot to filter one.

This is the one part of the system that cannot be made unattended without changing what it is. Everything upstream — fetching, ranking, tailoring, outreach drafting — runs on the server whether you are there or not. The applying does not, on purpose.

Target companies are drawn from the jobs you are actually applying to, so outreach follows the applications instead of running as a separate campaign. The work then splits in two: per company, work out and verify the address pattern; per person, find the named people worth writing to. Both halves are recorded, so nobody is written to twice.

• The address shape a company uses is a fact you establish once and reuse for everyone there, so the expensive part happens per company rather than per recipient.
• Addresses are checked for existence before a draft is ever created. Guessing an address and sending to it is how a personal domain acquires a reputation problem.
• The verification service is not perfectly honest — it drifts between answering truthfully, denying everything and confirming everything — so it is run alongside a known control and retried rather than trusted on a single answer.
• Nothing is sent on a schedule you did not start. The batch is named and counted before it goes, and re-counted if it changes while you are reading it.

Mail is sent from your own address using an app password: a token you generate yourself, that grants mail sending and nothing else, that is stored encrypted and shown back to you as four characters, and that you can revoke in one click without touching the rest of your account. The daily cap is twenty-five to thirty. That is not an API limit — a personal account sending outreach at volume stops looking like a person, and the cost of that is your own address.

This is the one stage of the six that can simply be left switched off. Everything upstream — fetching, ranking, tailoring, applying — works exactly the same without it.

Before the gateway, apply-service was reachable on a public address with no authentication at all: a plain request returned the stored name, email, phone and salary, and could drive the application queue. The mitigation was a cloud security list — which is to say, hoping nobody found the port. The gateway is what replaced that. Everything else moved onto an internal Docker network, and exactly one port on the box is now reachable from the internet.

A request arrives at Caddy over HTTPS, is proxied to the gateway, and passes four checks before any service sees it.

• Identity. A browser presents a cookie; the extension presents an opaque device token. Both resolve to an account, or to nobody — and a public path is allowed to be nobody.
• Rate limits. Two of them, and they are different problems: a general allowance sized at roughly triple the measured steady state of one person with a dashboard open, and a far tighter one on sign-in, which is the only unauthenticated endpoint here where guessing is worth anyone’s time.
• Header trust. Every inbound identity header is stripped, whatever it said and whether or not the caller turned out to be signed in, and the gateway’s own values are injected in their place.
• Routing. By path, in an order that is deliberately not alphabetical, with one circuit breaker per downstream service and a fallback response when a breaker is open.

Downstream services are told who the caller is by a header, and that is only sound if the header cannot have come from the caller. By default it absolutely can — nothing stops anyone adding an identity header to a request. So every request is wrapped before it is routed, and the wrapper strips first and injects second. Stripping only when authentication failed would leave the hole open on exactly the requests that matter.

X-User-Id      the account the gateway resolved
X-User-Email   its address
X-User-Role    admin, or not
X-Auth-Kind    ACCESS_TOKEN (a cookie) or DEVICE_TOKEN (the extension)

The other half of that guarantee is not in the gateway at all: header trust holds only while the downstream ports are unreachable from outside. If 8083 were still published, anyone could skip the gateway and assert whatever they liked directly. Nothing but the gateway is published, and that is a compose-file fact rather than a code one.

Two smaller things in the same file are worth naming, because both were silent failures that took days to find. Which proxies may be believed about the caller is an explicit pattern, not a wildcard — anything matching it can claim to be any address, which would make per-caller rate limiting meaningless and let an attacker forge their origin in the access log. And the gateway relays the client’s scheme untouched rather than appending its own view of it: it is not the component that knows whether the client used HTTPS, Caddy is, and a gateway that contradicts the edge makes every service behind it believe the deployment is unencrypted.

Passwords are stored as BCrypt hashes and never in any recoverable form. That is also why the sign-in rate limit is tight rather than generous: BCrypt is slow by design, so a guessing run is expensive for the server whether or not it ever finds a password.

• Session cookies are HttpOnly, so script on the page cannot read them; SameSite=Lax, so the browser does not attach them to a cross-site request; and Secure, which is only correct because there is TLS in front — a browser silently discards a Secure cookie delivered over plain HTTP.
• Device tokens are hashed with SHA-256 rather than BCrypt, and that is a deliberate difference from the password column. There is no guessing to slow down against 256 bits of randomness, and a deliberately slow hash on a five-second poll would be a cost paid for nothing.
• Signing out revokes the session row and records the still-valid access token, so a sign-out reaches a token that has not expired yet instead of leaving a fifteen-minute window open.
• CSRF protection is applied where the credential is ambient and not where it is not. A bearer or device token is attached only by code that deliberately read it out of storage; a token there would defend against nothing and would break the extension, which has no page to carry one.

The security chain denies by default. What is public is one explicit list — the sign-in endpoints, the recovery endpoints, the built front end and its routes — and everything absent from it requires a caller. Adding an endpoint does not quietly make it reachable. That list exists in two places, the gateway and the service, and a test asserts the two agree: they drifted once, and the stricter one silently won.

Credentials the system has to be able to use later — a mail app password, a per-account API key — are sealed with AES-256-GCM before they reach the database and opened on the way out. It is applied as a persistence-layer converter rather than as encrypt and decrypt calls in a service, because the failure mode of the second arrangement is one new code path that saves the plaintext, and that path would look entirely normal in review.

enc:v1:<base64( 12-byte IV ‖ AES-256-GCM ciphertext ‖ 16-byte tag )>

• GCM rather than CBC because it authenticates: a row edited by hand fails to open instead of decrypting into rubbish that gets used.
• The version prefix means the format can change without a migration that has to decrypt everything at once.
• The key is an environment variable of the application process. It appears in no row, so a copy of the data is not a copy of the secrets.

It is worth being equally clear about what this does not do. The process has to hand a plaintext password to a mail handshake, so it must be able to decrypt; anyone who can read the running container’s environment or its memory can read the secrets. Encryption at rest raises the bar from “one leaked file” to “a shell on the box”. The realistic leak is not an attacker with root — it is a database dump left somewhere, a volume snapshot, a backup copied to a laptop, a database restored onto a second box to debug something, one injection that reads a table. Those are the cheap and common attacks, and those are the ones this stops.

Per-account isolation is enforced in the queries themselves rather than in the UI. Four separate cross-account leaks were found and closed by making the account identifier part of the query rather than a filter applied afterwards — a list that is filtered after the fact is a list that has already been read.

Caddy terminates TLS with a certificate it obtains and renews by itself, for applynchill.com, a domain the project owns outright. Everything behind it speaks plain HTTP on an internal network, and Caddy is the component that tells them what the client actually used.

The access log is filtered before it is written: Cookie, Set-Cookie, Authorization and the extension’s device token are replaced rather than deleted, because knowing a header was present is often exactly what you are debugging. Declaring that filter replaces the proxy’s own default redactions rather than adding to them — which is a very easy way to start logging the thing you meant to hide.

Each Java service exposes exactly two management endpoints — health, and a metrics scrape — and nothing else. Environment, configuration properties, beans, loggers and heap dumps are not mapped at all: they answer as though they do not exist, rather than as though they exist and are refusing you.

Those two live on a second port that is not published in the compose file, so the only callers that can open a socket to them are the other containers on the internal network. That was not the obvious fix. The obvious fix — adding the metrics path to the public allowlist so the scraper could reach it — would have put a live metrics dump on the public internet, listing every endpoint and every counter, in order to make an internal scrape work.

• Moving the port is only half of it. The application’s security filter is registered on the management connector too, so the port change alone still refused the scraper — measured, not assumed. The second half is a security chain that matches on the port a connection arrived on, rather than on a path.
• It matches on the port and not on a path deliberately: a rule phrased as a path would be one edit away from opening those same endpoints on the public port, and that must not be possible whatever anyone later adds to an exposure list.
• The gateway exposes health only. Its own configuration holds the database password and the signing key, and it is the one process deliberately reachable from the internet.

• Latency percentiles need histogram buckets, not just counts and sums — with counts alone the percentile panel draws nothing at all. Buckets are enabled for the HTTP request timer specifically, because enabling them globally multiplies cardinality across every timer in the process.
• Sampling is one in ten and that is the right number here: the extension polls every five seconds forever, and the overwhelming majority of the traffic this service sees is an empty queue check. Tracing all of it would fill the backend with the least interesting request in the system.
• A second metrics exporter that ships with the tracing library is switched off. It defaults to on, and leaving it on means computing and pushing the same numbers a second time, to a destination nobody configured, once a minute, forever.

The dashboards were blank for a while, and the cause was a datasource reference rather than missing data. Querying the backend directly hides exactly that class of failure — the data was there the whole time, and the panel had no way to ask for it.

• A downstream service that is failing trips a circuit breaker at the gateway and returns a stated fallback rather than a stack trace or a hung connection. One breaker per service, not per route, so the low-traffic routes share a health signal with the busy ones instead of never seeing enough calls to notice.
• A board that rate-limits is a slower sweep. The window narrows, the pacing holds, and the jobs already collected are unaffected.
• A model that refuses leaves the job unenriched, and the next run continues with whatever is still empty. Nothing is written half-parsed.
• A tailored résumé that fails validation is discarded and the base résumé goes out. The application is never the thing that is sacrificed.
• A form field that cannot be answered from your profile stops the run and asks you. Nothing plausible is ever invented into an employer’s form.
• Any stage interrupted mid-run resumes from the database on the next start, because every stage writes its progress rather than holding it.

Roughly a hundred automated tests cover the services, and a good number of them exist because something in this list failed once in a way that was invisible: an allowlist that had drifted, a route predicate broader than the service behind it, a regex that compiled and matched nothing, a scheme header contradicted one hop too late. Those are the tests worth having.