Scanner Providers - Fighter90/career-ops-ui GitHub Wiki

Scanner Providers

As of v1.236.0 the scanner ships 94 adapters β€” 89 EN-region + 5 RU.

Keyword prefixes (v1.227.5). title_filter / content_filter entries default to case-insensitive substring matching, so a bare negative intern also rejects "International Product Manager". Two opt-in prefixes narrow one entry without changing that default: word:intern matches the whole word only, and stem:agent must start a word and may continue (separating "Agentforce" from "Reagents"). Boundaries are Unicode-aware, so they behave the same in accented and Cyrillic titles. A bare word: / stem: is treated as a typo and matches nothing β€” as a negative, an empty pattern would veto an entire scan. web-ui ignored both prefixes until v1.227.5, matching them as literal text. All scanning is in-process and zero-token: adapters hit public job-board APIs/feeds directly over HTTPS; no LLM is involved and no external script is spawned.

The two registries (read this first)

Adding an EN board touches two registries:

  1. Source meta registry β€” server/lib/sources/<slug>.mjs with an export const meta = { value, label, region, configKey? } block. server/lib/sources/registry.mjs readdirSync-scans the folder at boot and dynamically import()s every *.mjs (P-14, v1.69.0), so dropping a valid file auto-registers it (malformed meta β†’ skipped with one console.warn). This drives GET /api/scan/sources and the #/scan dropdown.
  2. Fetch-walk registry β€” server/lib/portals/adapters/<slug>.mjs plus its entry in server/lib/portals/registry.mjs::ALL_ADAPTERS (asserted === 89 with the exact sorted id list by tests/adapter-registry.test.mjs). Adapters implement matches()/buildEndpoint() (string-or-null contract) and the fetch/parse walk over tracked_companies: in portals.yml.

RU sources additionally need a RU_DISPATCH row in server/lib/ru-scanner.mjs, keyed by the meta's configKey.

EN adapters (89)

Kind: board-wide = one global feed searched by keyword Β· per-tenant = per-company slug/board (driven by tracked_companies:) Β· single-company = one fixed employer's careers backend.

id (value) Label Kind Notes
generalist-world Generalist World board-wide generalist/ops roles; fails loud on markup change
greenhouse Greenhouse per-tenant boards-api.greenhouse.io/v1/boards/<slug>/jobs
ashby Ashby per-tenant api.ashbyhq.com/posting-api/job-board/<slug>
lever Lever per-tenant api.lever.co/v0/postings/<slug>; v1.118.0: detects EU boards jobs.eu.lever.co β†’ api.eu.lever.co
pythonorg Python.org Jobs board-wide Python roles; employer attribution required
workable Workable per-tenant
smartrecruiters SmartRecruiters per-tenant
workday Workday per-tenant Reports workdayFallback in scan results; v1.119.0: browser-like headers (UA/accept-language/origin/referer derived from the CXS URL) clear Cloudflare-gated tenants
recruitee Recruitee per-tenant
teamtailor Teamtailor per-tenant Feed-based
personio Personio per-tenant An explicit personio: <slug> pins the tenant to <slug>.jobs.personio.de, for boards embedded as an iframe on a company domain where careers_url points nowhere useful. Anchored slug allowlist, still gated by the host regex; normalised through new URL() because web-ui's host check is case-sensitive β€” v1.237.0
breezy Breezy HR per-tenant
bamboohr BambooHR per-tenant
rippling Rippling per-tenant ats.rippling.com/<slug>
pinpoint Pinpoint per-tenant
comeet Comeet per-tenant careers-api URL
avature Avature per-tenant
successfactors SAP SuccessFactors (multi-brand RMK brand paths preserved β€” v1.125.1) per-tenant
jibeapply JibeApply (iCIMS) per-tenant Paginated (max 50 pages) β€” v1.117.0
softgarden softgarden per-tenant Cap 1000 jobs β€” v1.117.0
csod Cornerstone per-tenant 25/page pagination β€” v1.118.0
phenom Phenom per-tenant 100/page widget API β€” v1.118.0
radancy Radancy per-tenant Branded hosts carry no vendor token β†’ provider:-selected only, endpoint-shape checked β€” v1.118.0; legacy TalentBrew markup + JSON results-fragment transport β€” v1.130.0
beesite beesite (GJB) single-company jobs.mercedes-benz.com backend, 100/page β€” v1.117.0
deutschebahn Deutsche Bahn single-company db.jobs (anchored host regex) β€” v1.118.0
tkms TKMS single-company jobs.tkmsgroup.com β€” v1.118.0
hecklerkoch Heckler & Koch single-company v1.118.0
meituan Meituan single-company zhaopin.meituan.com JSON API; per-entry keywords/max_pages, empty-page retry β€” v1.119.0
tencent Tencent single-company careers.tencent.com Query API; per-entry keywords/max_pages β€” v1.119.0
oraclecloud Oracle Cloud (ORC) per-tenant recruitingCEJobRequisitions REST API of *.fa[.<region>][.ocs].oraclecloud.com tenants (JPMC, Oracle, BNY Mellon, Amex, …); siteNumber from careers_url, WAF-aware UA. A SHORT page no longer ends the walk: ORC filters rows server-side and serves short pages mid-list (Amex: TotalJobsCount 454 over pages of 200/199/54), so paging follows the reported total and only a tenant with no total stops on a short page; hasMore stays ignored and MAX_PAGES is the backstop β€” v1.237.0
wttj Welcome to the Jungle board-wide public Algolia index (wttj_jobs_production_en), rotating creds bootstrapped from /api/env; explicit wttj: { queries: [...] } config required β€” v1.124.0
agenticjobs Agentic Jobs board-wide agentic-engineering-jobs.com server-rendered listing, parsed in-process β€” v1.124.0
jobvite Jobvite per-tenant public XML feed app.jobvite.com/CompanyJobs/Xml.aspx?c=<companyEId>; companyEId from company_eid: config β†’ c= of an explicit api: β†’ board-page discovery. Two hosts SSRF-pinned (jobs.jobvite.com discovery + app.jobvite.com feed), no redirect followed. Migrated off the retired JSON API β€” v1.157.0
gem Gem per-tenant public GraphQL batch endpoint jobs.gem.com/api/public/graphql/batch, ?board= per company β€” v1.124.0
alibaba Alibaba single-company talent.alibaba.com/position/search JSON API (double-submit CSRF, no login); per-entry keywords/max_pages β€” v1.124.0
flowxtra Flowxtra board-wide no-auth cross-tenant aggregator app.flowxtra.com/api/central/jobs (paginated) β€” v1.129.1
vdab VDAB board-wide Flanders public employment service vindeenjob JSON API (www.vdab.be), keyword-driven, recall-first β€” v1.129.1
icims iCIMS per-tenant hosted-portal search pages careers-<tenant>.icims.com/jobs/search, auto-detected from any *.icims.com careers_url (distinct from jibeapply) β€” v1.129.1
a16z-speedrun-talent a16z Speedrun board-wide a16z Speedrun talent-network board feed; host-pinned speedrun-talent-network.com JSON API, HTTPS-only, 0-indexed pagination + page cap, per-company q/config threading β€” v1.130.0; adapter override host-pinned (exact host, HTTPS) with fallback to the canonical feed β€” v1.131.1
cryptocurrencyjobs Cryptocurrency Jobs board-wide Web3 board cryptocurrencyjobs.co via public RSS 2.0 (zero-auth); two-pass XML-entity decode, remote-only, employer from the "… at <Company>" title tail; parser cleanUrl uses the exact-match host guard (no subdomains) β€” v1.131.1
manfred getManfred board-wide Spanish/EU tech board with published salaries; host-pinned www.getmanfred.com/api/v2/public/offers JSON (zero-auth), single-request full catalogue, salary + remote-percentage parsing β€” v1.134.0
join JOIN per-company A company's JOIN board read from the Next.js __NEXT_DATA__ in join.com/companies/<slug> (host-pinned, page-capped, redirect:'error'); detected from a join.com careers URL β€” v1.135.0
getro Getro board-wide VC talent-network portfolio boards via api.getro.com/api/v2/collections/{id}/search/jobs POST (zero-auth); explicit numeric getro_collection:, newest-first pagination with age-bounded early stop, portfolio-employer attribution β€” v1.135.0
consider Consider board-wide getconsider.com VC portfolio boards via same-origin /api-boards/search-jobs POST; config-driven host pinned by a structural SSRF guard (public HTTPS only β€” rejects IP-literals/loopback/*.internal) β€” v1.135.0
joinup JOINUP board-wide Swiss startup board joinup.ch, SSR'd newest page of /browse/jobs (__NEXT_DATA__), fail-closed on a scraper break β€” v1.135.0
remotli Remotli board-wide remotli.ch curated Swiss remote board (CHF salaries) via ?remote=all JSON API; emits the employer's own ATS applyUrl so cross-listings dedup β€” v1.135.0
eightfold Eightfold per-tenant Eightfold AI talent-acquisition boards via https://<tenant>.eightfold.ai/api/apply/v2/jobs (zero-auth); host-pinned *.eightfold.ai (branded careers.<company>.com CNAME rejected), paginated with a safety cap, dead-board-throw, url-dedup β€” v1.136.0
rheinmetall Rheinmetall single-company v1.118.0
dassault Dassault Systèmes single-company Exalead feed
amazon Amazon single-company
ibm IBM single-company
arbeitsagentur Arbeitsagentur board-wide German federal job board
arbeitnow Arbeitnow board-wide
remoteok RemoteOK board-wide
remotive Remotive board-wide
weworkremotely We Work Remotely board-wide
workingnomads Working Nomads board-wide
nodesk NoDesk board-wide
himalayas Himalayas board-wide
jobicy Jobicy board-wide
jobspresso Jobspresso board-wide
4dayweek 4 Day Week board-wide
justjoin JustJoin.it board-wide Polish IT board
nofluffjobs NoFluffJobs board-wide Polish IT board
solidjobs SolidJobs board-wide Polish IT board
thehub The Hub board-wide Nordic startups
landingjobs Landing.jobs board-wide
getonbrd Get on Board board-wide LatAm
glints Glints board-wide SE Asia
jobstreet Jobstreet / SEEK board-wide SE Asia / ANZ / HK β€” SEEK v5 JobSearch API (/api/jobsearch/v5/search). One adapter, many markets: pick with siteKey β€” ID-Main (id.jobstreet.com), SG-Main, MY-Main, and HK-Main (hk.jobsdb.com β€” Hong Kong keeps the JobsDB brand on the same SEEK platform). Markets now also cover AU-Main (au.seek.com, formerly www.seek.com.au) and NZ-Main (nz.seek.com, formerly www.seek.co.nz) β€” SEEK is migrating hosts and both forms are allowlisted, because the transport refuses redirects (redirect:'error') and the old host would fail the day the API path 301s (v1.237.1). The detail path is keyed on the HOST: only id.jobstreet.com/*.jobstreet.co.id take the /id/ locale prefix β€” every other SEEK host 404s on /id/job/<id>, which is what made AU/NZ/SG/MY/HK links dead until v1.237.0. Opt-in appendWorkType suffixes titles with [Part time] β€” v1.237.0
themuse The Muse board-wide
jobbankca Job Bank (Canada) board-wide jobbank.gc.ca public ATOM feed (Canada's federal national employment service); host-pinned SSRF guard, config-driven keywords with a profile-target fallback β€” v1.212.0
mycareersfuture MyCareersFuture board-wide api.mycareersfuture.gov.sg public search API (Singapore's national job bank); POST + query-string pagination, host-pinned, config-driven keywords with a profile-target fallback β€” v1.213.0
larajobs LaraJobs board-wide RSS β€” v1.118.0
senjob Senjob board-wide senjob.com listing (Senegal β€” the scanner's first African source), HTML-parsed in-process, host-pinned; throws on a broken listing page rather than reporting an empty board β€” v1.210.0
yourator Yourator board-wide yourator.co/api/v4/jobs JSON API (Taiwan tech/digital); walks every page; employer ATS link emitted (display-only), API host-pinned β€” v1.211.0
torre Torre board-wide search.torre.co/opportunities/_search POST/JSON opportunity search (pan-LatAm remote roles, torre.ai); host-pinned HTTPS + redirect:error, single capped 20-row request (the API cannot page), configure search (+ experience) and optional remote_only β€” v1.221.0
itviec ITviec board-wide itviec.com HTML listing scrape (Vietnamese tech board); host-pinned HTTPS + per-company fail-soft; configure search/location β€” v1.227.5
careerviet CareerViet board-wide careerviet.vn HTML listing scrape (broad Vietnamese board); host-pinned HTTPS + page caps β€” v1.227.5
telegram Telegram board-wide Public channel preview at t.me/s/<channel>, configured in a top-level telegram_channels: block (not tracked_companies β€” a channel is not an employer and has no careers URL to detect). Host pinned to t.me exactly, HTTPS-only. Posts are PROSE, not job records: title = first substantive line, company/location only from explicit labels, unlabelled rows attributed to @channel rather than guessed; an empty parse throws instead of reading as "no vacancies". Lean on title_filter to drop ads and digests β€” v1.228.0
builtin Built In board-wide US tech board across its nine markets; employers post directly, so it aggregates like RemoteOK. Explicit provider: builtin only β€” a careers_url must never pull the whole board. Ships NO default query: an entry with neither queries: nor categories: scans nothing and says so. A market host outside the allowlist is refused, not swapped for the national board; each host must already be its own canonical form, and which form that is differs per board: builtin.com bare (its www. 301s away) but www.builtinseattle.com prefixed (the bare one 301s to it). resolveHost maps either spelling to the canonical one before a URL is built; assertHost then requires a fixed point, so a non-canonical host cannot reach the network at all β€” with redirect:'error' it would hard-fail anyway β€” v1.229.0
feishu-jobs Feishu Jobs per-tenant ι£žδΉ¦ζ‹›θ˜ β€” the endpoint the tenant's own careers page calls. Two host shapes only: jobs.bytedance.com exactly or *.jobs.feishu.cn (the leading dot stops jobs.feishu.cn.evil.test). The two use DIFFERENT job-page paths, so one URL template would give half the tenants dead links. Sends the macOS UA β€” the WAF 403s the Windows one β€” v1.229.0
garena Garena single company careers.garena.com, one fixed host. office shapes the job LINK, never the listing: every office code, invented ones included, returns the same board. ./.. are refused as URL segments outright, since encodeURIComponent leaves them intact as traversal β€” v1.229.0
mokahr MokaHR per-tenant ε›½ε†… HR SaaS. The response is an AES-128-CBC envelope whose key ships alongside the ciphertext β€” obfuscation, not security, but a plain JSON parse sees nothing, so decryption is part of reading the board. siteId reaches a request BODY, hence the strict tenant parse. Two tenants refused at config time: robots.txt excludes their careers path β€” v1.229.0
collage Collage per-tenant Collage HR's public job-site API. The path segment is the job-site address the tenant chose, not a slug derived from the company name, so it comes from api: or a secure.collage.co/jobs/<address> careers URL and is never guessed β€” a guessed address scans somebody else's board. Host AND path shape are pinned: without the path check, /login on api.collage.co would be accepted as a board β€” v1.231.0
telegram-channel Telegram (strict) per-channel The same t.me/s/ previews telegram reads, with the opposite trade: a post becomes a row ONLY when it names an employer (one of five explicit shapes) AND links to a vacancy page β€” never a homepage, listing root, form, shortener or social. Digests of several vacancies are dropped, because the scanner cannot split them. 32-77% of posts pass on HR-curated channels. provider:-selected only, since a bare t.me URL is ambiguous between the two sources. web-ui's port fixes a defect still present upstream: the location filter used ASCII \b, so it could not see Cyrillic and returned a city as the EMPLOYER β€” v1.231.0
higheredjobs HigherEdJobs board-wide RSS category feeds β€” v1.117.0
hackernews Hacker News (Who is hiring) board-wide Monthly thread
rss RSS generic Point it at any job RSS/Atom feed β€” the escape hatch when a board has no adapter

Querying the results (v1.228.2). GET /api/scan-results?q= matches search terms independently, combined with AND β€” ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ€ ΠΏΡ€ΠΎΠ΄ΡƒΠΊΡ‚Π° also finds "ΠœΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ€ ΠΏΠΎ Ρ€Π°Π·Π²ΠΈΡ‚ΠΈΡŽ ΠΏΡ€ΠΎΠ΄ΡƒΠΊΡ‚Π°", and word order does not matter. It was a single whole-phrase includes(q) until v1.228.2, which found only the exact adjacency: ΠΏΡ€ΠΎΠ΄Π°ΠΊΡ‚ ΠΌΠ΅Π½Π΅Π΄ΠΆΠ΅Ρ€ returned 32 rows where 162 carried both words. That is worth knowing when a search looks thin β€” job titles vary in wording far more than expected, so a short distinctive stem (ΠΏΡ€ΠΎΠ΄ΡƒΠΊΡ‚, product) beats a two-word phrase.

Telegram channels (v1.228.0)

Telegram is the one source configured outside tracked_companies, because a channel is not an employer β€” the scanner filters that list through detectApi(), and a channel has no careers URL to detect. expandTelegramChannels() in server/lib/en-scanner.mjs expands the block into ordinary adapter-selected entries, so quarantine, filters and dedup all run on one scan path rather than a parallel one.

telegram_channels:
  enabled: true
  max_posts: 100          # default cap per channel (hard cap 300)
  channels:
    - { name: "PHP jobs", channel: rabotaphp }
    - { name: "Salary PM", channel: salary_pm, max_posts: 50 }

channel: accepts rabotaphp, @rabotaphp, https://t.me/rabotaphp, or a link to a single post. Each channel is read from its public web preview at https://t.me/s/<channel> β€” plain server-rendered HTML, no bot token and no API key. (Telegram publishes no RSS, and the Bot API cannot read a channel your bot does not administer, so neither obvious route works.)

Three behaviours are deliberate and should not be "fixed":

  • A channel post is prose, not a job record. No structured fields exist. Separating real postings from ads and digests is your title_filter's job β€” the same one already tuned for other sources. These are leads, not listings.
  • The company is never guessed. With no explicit label the row is attributed to the channel (@rabotaphp); a wrong employer would enter the tracker as fact.
  • An empty parse is an error. t.me answers a private or missing channel with a redirect; returning zero rows would read as a quiet day and hide a typo in the config forever.

Parsing anchors on the three things a Telegram redesign is least likely to move: data-post="<channel>/<id>" (which both windows the page and carries the post's only stable URL), the message-text container, and the ISO <time datetime>.

RU sources (5)

Dispatched by ru-scanner.mjs::RU_DISPATCH and configured under russian_portals: in portals.yml (sources: ["hh", "habr", "trudvsem", "getmatch", "geekjob"] is the default).

id Label configKey Notes
hh.ru hh.ru hh HTML scrape of hh.ru/search/vacancy (the JSON API 403s programmatic clients)
habr-career Habr Career habr career.habr.com/api/...
trudvsem Trudvsem trudvsem Russian federal job board
getmatch GetMatch getmatch
geekjob GeekJob geekjob

How to add a new provider

The full walkthrough lives in the in-app help guide Β§17 ("How to add a new job-portal source"). Summary:

  1. Create server/lib/sources/<slug>.mjs exporting meta = { value, label, region: 'en'|'ru', configKey? } and a search<Name>(query, opts) returning normalized job objects: { id, title, company, url, salary, location, isRemote, workplaceType, relocates, date, snippet, source }. It auto-registers at boot β€” no registry edit.

  2. EN boards: add server/lib/portals/adapters/<slug>.mjs implementing the adapter contract β€” matches(url) and buildEndpoint(...) (must return a string or null, never an object) β€” and register it in server/lib/portals/registry.mjs::ALL_ADAPTERS. Update the count assertions in tests/adapter-registry.test.mjs and tests/scan-sources-endpoint.test.mjs.

  3. RU sources: add a RU_DISPATCH row in ru-scanner.mjs.

  4. SSRF host-pinning (mandatory): every adapter pins its hostname with an anchored regex (e.g. /(^|\.)justjoin\.it$/i β€” so justjoin.it.evil.com can't spoof) and rejects http:. Adapters with no stable vendor host (e.g. Radancy) validate the endpoint path shape and are selected only via an explicit provider: key in portals.yml. Parity tests must cover: evil host rejected, http: rejected, suffix-spoof rejected, parse happy path, pagination stop condition.

  5. Host-controlled ids go through _safe-url.mjs (since v1.234.0): encodeURIComponent throws URIError on a lone UTF-16 surrogate, and a JSON payload can carry one β€” inside a parse loop that aborts the loop and loses the whole page. Build the job URL with safeEncodeURIComponent(id) and drop just that posting on null. Config-derived values (portals.yml segments, locale codes) keep the strict encode so a config bug fails loud. tests/sources-url-encoding-surrogate.test.mjs enforces it: a new source with a bare encodeURIComponent on a url line fails CI unless it is allowlisted there with a reason.

  6. Identity in a URL fragment must be promoted (since v1.236.0): the dedup key (server/lib/url-key.mjs) drops fragments, because almost all of them are presentation-only. If a board puts the posting id ONLY in the hash β€” MokaHR's …/{tenant}/{id}#/job/{n} is the case in our registry β€” every job on that tenant collapses to one key and the extras vanish silently (from the pipeline, and from the scanner as "already seen"). promoteKnownFragmentIdentity recognizes #/job/{id} and #/jobs/{id}; a board with a different hash shape needs a case added there and a test, not a change to what the key drops.

  7. A locale prefix in a detail path must be keyed on the HOST (since v1.237.0): Jobstreet/SEEK is one adapter over many market hosts, and only the Indonesian ones serve /id/job/<id> β€” /id/ is a locale prefix, not part of the path. Every other SEEK host (www.seek.com.au, www.seek.co.nz, sg./my.jobstreet.com, hk.jobsdb.com) answers 404 on it. Because those hosts were already in the allowlist, the scan returned their postings and every one pointed at a dead page β€” validation passes, the link is still broken. ID_LOCALE_HOSTS + jobDetailPath(origin) now decide per host, and an unparseable origin falls through to the common path rather than throwing. Two of our own tests had encoded the bug by asserting the 404 form; when a fix makes an existing assertion fail, check which of the two is wrong before "repairing" the test.

  8. A short page is not the end of a list (since v1.237.0): Oracle Cloud filters rows server-side, so it serves fewer rows than the limit in the middle of a board. Measured on American Express: TotalJobsCount 454 over pages of 200 / 199 / 54 β€” treating the 199 as the end dropped the last 54 postings, 12% of the board. An empty page always ends the walk; when the tenant reports a total, page until offset + PAGE_SIZE >= total; only a tenant that reports no total may stop on a short page. This follows the wider convention that an API "may return fewer results than the number requested … even if not at the end of the collection" (Google AIP-158). Note the consequence: once short pages stop ending the walk, the page cap is the only backstop against a tenant reporting a bogus total, so it needs its own test.

  9. Config-driven providers: POST/GraphQL/paginated boards thread opts.company through buildEndpoint; keep buildEndpoint returning a string.

  10. Add a parity test suite (tests/sources-parity-*.test.mjs, stubbed transport β€” no network), bump the registry counts, and fan out docs Γ—17 per the Release Process.

Scan behavior

  • Entry point: GET /api/stream/scan?source=ats|regional|both (SSE, one done per phase, final:true on the last).
  • dryRun=1 skips writes to data/scan-history.tsv / data/last-scan.json; company=<slug> narrows the ATS walk.
  • Results are deduped against data/scan-history.tsv, sanitized (scan-sanitize.mjs), quarantine-filtered (scan-quarantine.mjs), location-filtered, and matched against your target roles / two-pager (β—Ž badge).
  • Client disconnect aborts in-flight upstream fetches (AbortSignal).

Since v1.119.1 the #/scan Source dropdown's offline fallback (FALLBACK_SOURCES in public/js/views/scan.js) is gated by tests/scan-fallback-sources.test.mjs β€” exact value+label parity with the source registry, so a new adapter can't ship without the client list.

⚠️ **GitHub.com Fallback** ⚠️