six stages · fetch-service · port 8081
Fetch
Sweeps real ATS boards for the roles you have asked for, on a rolling window, and writes what it finds into one shared corpus.
How this stage flows
The roles you have set and the track packs that go with them are planned into a sweep, one per role per board. fetch-service paces that sweep and calls four kinds of source over HTTP: Greenhouse, Lever, the Ashby, iCIMS and Adzuna listing APIs, and company career pages. What comes back is deduplicated and written into one shared corpus.
a sweep is planned per role, per board
HTTP, paced
deduplicated, then stored
The sweep is the only stage that talks to the outside world on a schedule. Everything after it works from the corpus.
A sweep is planned rather than run flat out. Each of your target roles is paired with the boards that role actually appears on, and the pairs are worked through a fixed thread pool with a cap on how many run for any one role. The cost of a sweep is linear in the number of roles, which is why the role list is a list you edit rather than a box you type anything into.
The rolling window
A normal run takes the last twenty-four hours. A first run, or one that finds a gap in the record, widens to seven days and backfills it. The distinction decides how much of a board gets re-read, and a top-up that behaves like a backfill is a sweep that will get the address rate-limited.
Undated postings are aged from when they were first seen rather than assumed fresh. A board that omits a date is common; a pipeline that treats every such posting as new is a pipeline that applies to the same job every day.
Why the same job does not arrive twice
Deduplication is a fingerprint of company and role, not the URL. URLs were the obvious choice and the wrong one: several boards mint a fresh tracking link per session, so the same requisition arrives under a new address every sweep. Location is deliberately left out of the fingerprint, because one role posted in four cities is one job.
What it costs
Nothing but time and politeness. Fetch calls no model and spends no key. The only real limit is the boards themselves: swept without pacing, at least one of them will block the address for a while, and the symptom looks exactly like every company delisting at once.
Found this useful? Add applyNchill as a preferred source on Google