Scanner Providers - Fighter90/career-ops-ui GitHub Wiki

Scanner Providers

As of v1.228.0 the scanner ships 86 adapters — 81 EN-region + 5 RU.

Keyword prefixes (v1.227.5). title_filter / content_filter entries default to case-insensitive substring matching, so a bare negative intern also rejects "International Product Manager". Two opt-in prefixes narrow one entry without changing that default: word:intern matches the whole word only, and stem:agent must start a word and may continue (separating "Agentforce" from "Reagents"). Boundaries are Unicode-aware, so they behave the same in accented and Cyrillic titles. A bare word: / stem: is treated as a typo and matches nothing — as a negative, an empty pattern would veto an entire scan. web-ui ignored both prefixes until v1.227.5, matching them as literal text. All scanning is in-process and zero-token: adapters hit public job-board APIs/feeds directly over HTTPS; no LLM is involved and no external script is spawned.

The two registries (read this first)

Adding an EN board touches two registries:

  1. Source meta registryserver/lib/sources/<slug>.mjs with an export const meta = { value, label, region, configKey? } block. server/lib/sources/registry.mjs readdirSync-scans the folder at boot and dynamically import()s every *.mjs (P-14, v1.69.0), so dropping a valid file auto-registers it (malformed meta → skipped with one console.warn). This drives GET /api/scan/sources and the #/scan dropdown.
  2. Fetch-walk registryserver/lib/portals/adapters/<slug>.mjs plus its entry in server/lib/portals/registry.mjs::ALL_ADAPTERS (asserted === 81 with the exact sorted id list by tests/adapter-registry.test.mjs). Adapters implement matches()/buildEndpoint() (string-or-null contract) and the fetch/parse walk over tracked_companies: in portals.yml.

RU sources additionally need a RU_DISPATCH row in server/lib/ru-scanner.mjs, keyed by the meta's configKey.

EN adapters (81)

Kind: board-wide = one global feed searched by keyword · per-tenant = per-company slug/board (driven by tracked_companies:) · single-company = one fixed employer's careers backend.

id (value) Label Kind Notes
greenhouse Greenhouse per-tenant boards-api.greenhouse.io/v1/boards/<slug>/jobs
ashby Ashby per-tenant api.ashbyhq.com/posting-api/job-board/<slug>
lever Lever per-tenant api.lever.co/v0/postings/<slug>; v1.118.0: detects EU boards jobs.eu.lever.coapi.eu.lever.co
workable Workable per-tenant
smartrecruiters SmartRecruiters per-tenant
workday Workday per-tenant Reports workdayFallback in scan results; v1.119.0: browser-like headers (UA/accept-language/origin/referer derived from the CXS URL) clear Cloudflare-gated tenants
recruitee Recruitee per-tenant
teamtailor Teamtailor per-tenant Feed-based
personio Personio per-tenant
breezy Breezy HR per-tenant
bamboohr BambooHR per-tenant
rippling Rippling per-tenant ats.rippling.com/<slug>
pinpoint Pinpoint per-tenant
comeet Comeet per-tenant careers-api URL
avature Avature per-tenant
successfactors SAP SuccessFactors (multi-brand RMK brand paths preserved — v1.125.1) per-tenant
jibeapply JibeApply (iCIMS) per-tenant Paginated (max 50 pages) — v1.117.0
softgarden softgarden per-tenant Cap 1000 jobs — v1.117.0
csod Cornerstone per-tenant 25/page pagination — v1.118.0
phenom Phenom per-tenant 100/page widget API — v1.118.0
radancy Radancy per-tenant Branded hosts carry no vendor token → provider:-selected only, endpoint-shape checked — v1.118.0; legacy TalentBrew markup + JSON results-fragment transport — v1.130.0
beesite beesite (GJB) single-company jobs.mercedes-benz.com backend, 100/page — v1.117.0
deutschebahn Deutsche Bahn single-company db.jobs (anchored host regex) — v1.118.0
tkms TKMS single-company jobs.tkmsgroup.com — v1.118.0
hecklerkoch Heckler & Koch single-company v1.118.0
meituan Meituan single-company zhaopin.meituan.com JSON API; per-entry keywords/max_pages, empty-page retry — v1.119.0
tencent Tencent single-company careers.tencent.com Query API; per-entry keywords/max_pages — v1.119.0
oraclecloud Oracle Cloud (ORC) per-tenant recruitingCEJobRequisitions REST API of *.fa[.<region>][.ocs].oraclecloud.com tenants (JPMC, Oracle, BNY Mellon, Amex, …); siteNumber from careers_url, WAF-aware UA — v1.123.0
wttj Welcome to the Jungle board-wide public Algolia index (wttj_jobs_production_en), rotating creds bootstrapped from /api/env; explicit wttj: { queries: [...] } config required — v1.124.0
agenticjobs Agentic Jobs board-wide agentic-engineering-jobs.com server-rendered listing, parsed in-process — v1.124.0
jobvite Jobvite per-tenant public XML feed app.jobvite.com/CompanyJobs/Xml.aspx?c=<companyEId>; companyEId from company_eid: config → c= of an explicit api: → board-page discovery. Two hosts SSRF-pinned (jobs.jobvite.com discovery + app.jobvite.com feed), no redirect followed. Migrated off the retired JSON API — v1.157.0
gem Gem per-tenant public GraphQL batch endpoint jobs.gem.com/api/public/graphql/batch, ?board= per company — v1.124.0
alibaba Alibaba single-company talent.alibaba.com/position/search JSON API (double-submit CSRF, no login); per-entry keywords/max_pages — v1.124.0
flowxtra Flowxtra board-wide no-auth cross-tenant aggregator app.flowxtra.com/api/central/jobs (paginated) — v1.129.1
vdab VDAB board-wide Flanders public employment service vindeenjob JSON API (www.vdab.be), keyword-driven, recall-first — v1.129.1
icims iCIMS per-tenant hosted-portal search pages careers-<tenant>.icims.com/jobs/search, auto-detected from any *.icims.com careers_url (distinct from jibeapply) — v1.129.1
a16z-speedrun-talent a16z Speedrun board-wide a16z Speedrun talent-network board feed; host-pinned speedrun-talent-network.com JSON API, HTTPS-only, 0-indexed pagination + page cap, per-company q/config threading — v1.130.0; adapter override host-pinned (exact host, HTTPS) with fallback to the canonical feed — v1.131.1
cryptocurrencyjobs Cryptocurrency Jobs board-wide Web3 board cryptocurrencyjobs.co via public RSS 2.0 (zero-auth); two-pass XML-entity decode, remote-only, employer from the "… at <Company>" title tail; parser cleanUrl uses the exact-match host guard (no subdomains) — v1.131.1
manfred getManfred board-wide Spanish/EU tech board with published salaries; host-pinned www.getmanfred.com/api/v2/public/offers JSON (zero-auth), single-request full catalogue, salary + remote-percentage parsing — v1.134.0
join JOIN per-company A company's JOIN board read from the Next.js __NEXT_DATA__ in join.com/companies/<slug> (host-pinned, page-capped, redirect:'error'); detected from a join.com careers URL — v1.135.0
getro Getro board-wide VC talent-network portfolio boards via api.getro.com/api/v2/collections/{id}/search/jobs POST (zero-auth); explicit numeric getro_collection:, newest-first pagination with age-bounded early stop, portfolio-employer attribution — v1.135.0
consider Consider board-wide getconsider.com VC portfolio boards via same-origin /api-boards/search-jobs POST; config-driven host pinned by a structural SSRF guard (public HTTPS only — rejects IP-literals/loopback/*.internal) — v1.135.0
joinup JOINUP board-wide Swiss startup board joinup.ch, SSR'd newest page of /browse/jobs (__NEXT_DATA__), fail-closed on a scraper break — v1.135.0
remotli Remotli board-wide remotli.ch curated Swiss remote board (CHF salaries) via ?remote=all JSON API; emits the employer's own ATS applyUrl so cross-listings dedup — v1.135.0
eightfold Eightfold per-tenant Eightfold AI talent-acquisition boards via https://<tenant>.eightfold.ai/api/apply/v2/jobs (zero-auth); host-pinned *.eightfold.ai (branded careers.<company>.com CNAME rejected), paginated with a safety cap, dead-board-throw, url-dedup — v1.136.0
rheinmetall Rheinmetall single-company v1.118.0
dassault Dassault Systèmes single-company Exalead feed
amazon Amazon single-company
ibm IBM single-company
arbeitsagentur Arbeitsagentur board-wide German federal job board
arbeitnow Arbeitnow board-wide
remoteok RemoteOK board-wide
remotive Remotive board-wide
weworkremotely We Work Remotely board-wide
workingnomads Working Nomads board-wide
nodesk NoDesk board-wide
himalayas Himalayas board-wide
jobicy Jobicy board-wide
jobspresso Jobspresso board-wide
4dayweek 4 Day Week board-wide
justjoin JustJoin.it board-wide Polish IT board
nofluffjobs NoFluffJobs board-wide Polish IT board
solidjobs SolidJobs board-wide Polish IT board
thehub The Hub board-wide Nordic startups
landingjobs Landing.jobs board-wide
getonbrd Get on Board board-wide LatAm
glints Glints board-wide SE Asia
jobstreet Jobstreet / SEEK board-wide SE Asia / ANZ / HK — SEEK v5 JobSearch API (/api/jobsearch/v5/search). One adapter, many markets: pick with siteKeyID-Main (id.jobstreet.com), SG-Main, MY-Main, and HK-Main (hk.jobsdb.com — Hong Kong keeps the JobsDB brand on the same SEEK platform). Migrated off the retired chalice-search v4 endpoint — v1.227.5
themuse The Muse board-wide
jobbankca Job Bank (Canada) board-wide jobbank.gc.ca public ATOM feed (Canada's federal national employment service); host-pinned SSRF guard, config-driven keywords with a profile-target fallback — v1.212.0
mycareersfuture MyCareersFuture board-wide api.mycareersfuture.gov.sg public search API (Singapore's national job bank); POST + query-string pagination, host-pinned, config-driven keywords with a profile-target fallback — v1.213.0
larajobs LaraJobs board-wide RSS — v1.118.0
senjob Senjob board-wide senjob.com listing (Senegal — the scanner's first African source), HTML-parsed in-process, host-pinned; throws on a broken listing page rather than reporting an empty board — v1.210.0
yourator Yourator board-wide yourator.co/api/v4/jobs JSON API (Taiwan tech/digital); walks every page; employer ATS link emitted (display-only), API host-pinned — v1.211.0
torre Torre board-wide search.torre.co/opportunities/_search POST/JSON opportunity search (pan-LatAm remote roles, torre.ai); host-pinned HTTPS + redirect:error, single capped 20-row request (the API cannot page), configure search (+ experience) and optional remote_only — v1.221.0
itviec ITviec board-wide itviec.com HTML listing scrape (Vietnamese tech board); host-pinned HTTPS + per-company fail-soft; configure search/location — v1.227.5
careerviet CareerViet board-wide careerviet.vn HTML listing scrape (broad Vietnamese board); host-pinned HTTPS + page caps — v1.227.5
telegram Telegram board-wide Public channel preview at t.me/s/<channel>, configured in a top-level telegram_channels: block (not tracked_companies — a channel is not an employer and has no careers URL to detect). Host pinned to t.me exactly, HTTPS-only. Posts are PROSE, not job records: title = first substantive line, company/location only from explicit labels, unlabelled rows attributed to @channel rather than guessed; an empty parse throws instead of reading as "no vacancies". Lean on title_filter to drop ads and digests — v1.228.0
higheredjobs HigherEdJobs board-wide RSS category feeds — v1.117.0
hackernews Hacker News (Who is hiring) board-wide Monthly thread
rss RSS generic Point it at any job RSS/Atom feed — the escape hatch when a board has no adapter

Querying the results (v1.228.2). GET /api/scan-results?q= matches search terms independently, combined with AND — менеджер продукта also finds "Менеджер по развитию продукта", and word order does not matter. It was a single whole-phrase includes(q) until v1.228.2, which found only the exact adjacency: продакт менеджер returned 32 rows where 162 carried both words. That is worth knowing when a search looks thin — job titles vary in wording far more than expected, so a short distinctive stem (продукт, product) beats a two-word phrase.

Telegram channels (v1.228.0)

Telegram is the one source configured outside tracked_companies, because a channel is not an employer — the scanner filters that list through detectApi(), and a channel has no careers URL to detect. expandTelegramChannels() in server/lib/en-scanner.mjs expands the block into ordinary adapter-selected entries, so quarantine, filters and dedup all run on one scan path rather than a parallel one.

telegram_channels:
  enabled: true
  max_posts: 100          # default cap per channel (hard cap 300)
  channels:
    - { name: "PHP jobs", channel: rabotaphp }
    - { name: "Salary PM", channel: salary_pm, max_posts: 50 }

channel: accepts rabotaphp, @rabotaphp, https://t.me/rabotaphp, or a link to a single post. Each channel is read from its public web preview at https://t.me/s/<channel> — plain server-rendered HTML, no bot token and no API key. (Telegram publishes no RSS, and the Bot API cannot read a channel your bot does not administer, so neither obvious route works.)

Three behaviours are deliberate and should not be "fixed":

  • A channel post is prose, not a job record. No structured fields exist. Separating real postings from ads and digests is your title_filter's job — the same one already tuned for other sources. These are leads, not listings.
  • The company is never guessed. With no explicit label the row is attributed to the channel (@rabotaphp); a wrong employer would enter the tracker as fact.
  • An empty parse is an error. t.me answers a private or missing channel with a redirect; returning zero rows would read as a quiet day and hide a typo in the config forever.

Parsing anchors on the three things a Telegram redesign is least likely to move: data-post="<channel>/<id>" (which both windows the page and carries the post's only stable URL), the message-text container, and the ISO <time datetime>.

RU sources (5)

Dispatched by ru-scanner.mjs::RU_DISPATCH and configured under russian_portals: in portals.yml (sources: ["hh", "habr", "trudvsem", "getmatch", "geekjob"] is the default).

id Label configKey Notes
hh.ru hh.ru hh HTML scrape of hh.ru/search/vacancy (the JSON API 403s programmatic clients)
habr-career Habr Career habr career.habr.com/api/...
trudvsem Trudvsem trudvsem Russian federal job board
getmatch GetMatch getmatch
geekjob GeekJob geekjob

How to add a new provider

The full walkthrough lives in the in-app help guide §17 ("How to add a new job-portal source"). Summary:

  1. Create server/lib/sources/<slug>.mjs exporting meta = { value, label, region: 'en'|'ru', configKey? } and a search<Name>(query, opts) returning normalized job objects: { id, title, company, url, salary, location, isRemote, workplaceType, relocates, date, snippet, source }. It auto-registers at boot — no registry edit.
  2. EN boards: add server/lib/portals/adapters/<slug>.mjs implementing the adapter contract — matches(url) and buildEndpoint(...) (must return a string or null, never an object) — and register it in server/lib/portals/registry.mjs::ALL_ADAPTERS. Update the count assertions in tests/adapter-registry.test.mjs and tests/scan-sources-endpoint.test.mjs.
  3. RU sources: add a RU_DISPATCH row in ru-scanner.mjs.
  4. SSRF host-pinning (mandatory): every adapter pins its hostname with an anchored regex (e.g. /(^|\.)justjoin\.it$/i — so justjoin.it.evil.com can't spoof) and rejects http:. Adapters with no stable vendor host (e.g. Radancy) validate the endpoint path shape and are selected only via an explicit provider: key in portals.yml. Parity tests must cover: evil host rejected, http: rejected, suffix-spoof rejected, parse happy path, pagination stop condition.
  5. Config-driven providers: POST/GraphQL/paginated boards thread opts.company through buildEndpoint; keep buildEndpoint returning a string.
  6. Add a parity test suite (tests/sources-parity-*.test.mjs, stubbed transport — no network), bump the registry counts, and fan out docs ×17 per the Release Process.

Scan behavior

  • Entry point: GET /api/stream/scan?source=ats|regional|both (SSE, one done per phase, final:true on the last).
  • dryRun=1 skips writes to data/scan-history.tsv / data/last-scan.json; company=<slug> narrows the ATS walk.
  • Results are deduped against data/scan-history.tsv, sanitized (scan-sanitize.mjs), quarantine-filtered (scan-quarantine.mjs), location-filtered, and matched against your target roles / two-pager ( badge).
  • Client disconnect aborts in-flight upstream fetches (AbortSignal).

Since v1.119.1 the #/scan Source dropdown's offline fallback (FALLBACK_SOURCES in public/js/views/scan.js) is gated by tests/scan-fallback-sources.test.mjs — exact value+label parity with the source registry, so a new adapter can't ship without the client list.

⚠️ **GitHub.com Fallback** ⚠️