Scanner Providers - Fighter90/career-ops-ui GitHub Wiki
As of v1.228.0 the scanner ships 86 adapters — 81 EN-region + 5 RU.
Keyword prefixes (v1.227.5).
title_filter/content_filterentries default to case-insensitive substring matching, so a bare negativeinternalso rejects "International Product Manager". Two opt-in prefixes narrow one entry without changing that default:word:internmatches the whole word only, andstem:agentmust start a word and may continue (separating "Agentforce" from "Reagents"). Boundaries are Unicode-aware, so they behave the same in accented and Cyrillic titles. A bareword:/stem:is treated as a typo and matches nothing — as a negative, an empty pattern would veto an entire scan. web-ui ignored both prefixes until v1.227.5, matching them as literal text. All scanning is in-process and zero-token: adapters hit public job-board APIs/feeds directly over HTTPS; no LLM is involved and no external script is spawned.
Adding an EN board touches two registries:
-
Source meta registry —
server/lib/sources/<slug>.mjswith anexport const meta = { value, label, region, configKey? }block.server/lib/sources/registry.mjsreaddirSync-scans the folder at boot and dynamicallyimport()s every*.mjs(P-14, v1.69.0), so dropping a valid file auto-registers it (malformedmeta→ skipped with oneconsole.warn). This drivesGET /api/scan/sourcesand the#/scandropdown. -
Fetch-walk registry —
server/lib/portals/adapters/<slug>.mjsplus its entry inserver/lib/portals/registry.mjs::ALL_ADAPTERS(asserted=== 81with the exact sorted id list bytests/adapter-registry.test.mjs). Adapters implementmatches()/buildEndpoint()(string-or-null contract) and the fetch/parse walk overtracked_companies:inportals.yml.
RU sources additionally need a RU_DISPATCH row in server/lib/ru-scanner.mjs, keyed by the meta's configKey.
Kind: board-wide = one global feed searched by keyword · per-tenant = per-company slug/board (driven by tracked_companies:) · single-company = one fixed employer's careers backend.
id (value) |
Label | Kind | Notes |
|---|---|---|---|
greenhouse |
Greenhouse | per-tenant | boards-api.greenhouse.io/v1/boards/<slug>/jobs |
ashby |
Ashby | per-tenant | api.ashbyhq.com/posting-api/job-board/<slug> |
lever |
Lever | per-tenant |
api.lever.co/v0/postings/<slug>; v1.118.0: detects EU boards jobs.eu.lever.co → api.eu.lever.co
|
workable |
Workable | per-tenant | |
smartrecruiters |
SmartRecruiters | per-tenant | |
workday |
Workday | per-tenant | Reports workdayFallback in scan results; v1.119.0: browser-like headers (UA/accept-language/origin/referer derived from the CXS URL) clear Cloudflare-gated tenants |
recruitee |
Recruitee | per-tenant | |
teamtailor |
Teamtailor | per-tenant | Feed-based |
personio |
Personio | per-tenant | |
breezy |
Breezy HR | per-tenant | |
bamboohr |
BambooHR | per-tenant | |
rippling |
Rippling | per-tenant | ats.rippling.com/<slug> |
pinpoint |
Pinpoint | per-tenant | |
comeet |
Comeet | per-tenant | careers-api URL |
avature |
Avature | per-tenant | |
successfactors |
SAP SuccessFactors (multi-brand RMK brand paths preserved — v1.125.1) | per-tenant | |
jibeapply |
JibeApply (iCIMS) | per-tenant | Paginated (max 50 pages) — v1.117.0 |
softgarden |
softgarden | per-tenant | Cap 1000 jobs — v1.117.0 |
csod |
Cornerstone | per-tenant | 25/page pagination — v1.118.0 |
phenom |
Phenom | per-tenant | 100/page widget API — v1.118.0 |
radancy |
Radancy | per-tenant | Branded hosts carry no vendor token → provider:-selected only, endpoint-shape checked — v1.118.0; legacy TalentBrew markup + JSON results-fragment transport — v1.130.0 |
beesite |
beesite (GJB) | single-company | jobs.mercedes-benz.com backend, 100/page — v1.117.0 |
deutschebahn |
Deutsche Bahn | single-company |
db.jobs (anchored host regex) — v1.118.0 |
tkms |
TKMS | single-company |
jobs.tkmsgroup.com — v1.118.0 |
hecklerkoch |
Heckler & Koch | single-company | v1.118.0 |
meituan |
Meituan | single-company | zhaopin.meituan.com JSON API; per-entry keywords/max_pages, empty-page retry — v1.119.0 |
tencent |
Tencent | single-company | careers.tencent.com Query API; per-entry keywords/max_pages — v1.119.0 |
oraclecloud |
Oracle Cloud (ORC) | per-tenant |
recruitingCEJobRequisitions REST API of *.fa[.<region>][.ocs].oraclecloud.com tenants (JPMC, Oracle, BNY Mellon, Amex, …); siteNumber from careers_url, WAF-aware UA — v1.123.0 |
wttj |
Welcome to the Jungle | board-wide | public Algolia index (wttj_jobs_production_en), rotating creds bootstrapped from /api/env; explicit wttj: { queries: [...] } config required — v1.124.0 |
agenticjobs |
Agentic Jobs | board-wide | agentic-engineering-jobs.com server-rendered listing, parsed in-process — v1.124.0 |
jobvite |
Jobvite | per-tenant | public XML feed app.jobvite.com/CompanyJobs/Xml.aspx?c=<companyEId>; companyEId from company_eid: config → c= of an explicit api: → board-page discovery. Two hosts SSRF-pinned (jobs.jobvite.com discovery + app.jobvite.com feed), no redirect followed. Migrated off the retired JSON API — v1.157.0 |
gem |
Gem | per-tenant | public GraphQL batch endpoint jobs.gem.com/api/public/graphql/batch, ?board= per company — v1.124.0 |
alibaba |
Alibaba | single-company |
talent.alibaba.com/position/search JSON API (double-submit CSRF, no login); per-entry keywords/max_pages — v1.124.0 |
flowxtra |
Flowxtra | board-wide | no-auth cross-tenant aggregator app.flowxtra.com/api/central/jobs (paginated) — v1.129.1 |
vdab |
VDAB | board-wide | Flanders public employment service vindeenjob JSON API (www.vdab.be), keyword-driven, recall-first — v1.129.1 |
icims |
iCIMS | per-tenant | hosted-portal search pages careers-<tenant>.icims.com/jobs/search, auto-detected from any *.icims.com careers_url (distinct from jibeapply) — v1.129.1 |
a16z-speedrun-talent |
a16z Speedrun | board-wide | a16z Speedrun talent-network board feed; host-pinned speedrun-talent-network.com JSON API, HTTPS-only, 0-indexed pagination + page cap, per-company q/config threading — v1.130.0; adapter override host-pinned (exact host, HTTPS) with fallback to the canonical feed — v1.131.1 |
cryptocurrencyjobs |
Cryptocurrency Jobs | board-wide | Web3 board cryptocurrencyjobs.co via public RSS 2.0 (zero-auth); two-pass XML-entity decode, remote-only, employer from the "… at <Company>" title tail; parser cleanUrl uses the exact-match host guard (no subdomains) — v1.131.1 |
manfred |
getManfred | board-wide | Spanish/EU tech board with published salaries; host-pinned www.getmanfred.com/api/v2/public/offers JSON (zero-auth), single-request full catalogue, salary + remote-percentage parsing — v1.134.0 |
join |
JOIN | per-company | A company's JOIN board read from the Next.js __NEXT_DATA__ in join.com/companies/<slug> (host-pinned, page-capped, redirect:'error'); detected from a join.com careers URL — v1.135.0 |
getro |
Getro | board-wide | VC talent-network portfolio boards via api.getro.com/api/v2/collections/{id}/search/jobs POST (zero-auth); explicit numeric getro_collection:, newest-first pagination with age-bounded early stop, portfolio-employer attribution — v1.135.0 |
consider |
Consider | board-wide | getconsider.com VC portfolio boards via same-origin /api-boards/search-jobs POST; config-driven host pinned by a structural SSRF guard (public HTTPS only — rejects IP-literals/loopback/*.internal) — v1.135.0 |
joinup |
JOINUP | board-wide | Swiss startup board joinup.ch, SSR'd newest page of /browse/jobs (__NEXT_DATA__), fail-closed on a scraper break — v1.135.0 |
remotli |
Remotli | board-wide |
remotli.ch curated Swiss remote board (CHF salaries) via ?remote=all JSON API; emits the employer's own ATS applyUrl so cross-listings dedup — v1.135.0 |
eightfold |
Eightfold | per-tenant | Eightfold AI talent-acquisition boards via https://<tenant>.eightfold.ai/api/apply/v2/jobs (zero-auth); host-pinned *.eightfold.ai (branded careers.<company>.com CNAME rejected), paginated with a safety cap, dead-board-throw, url-dedup — v1.136.0 |
rheinmetall |
Rheinmetall | single-company | v1.118.0 |
dassault |
Dassault Systèmes | single-company | Exalead feed |
amazon |
Amazon | single-company | |
ibm |
IBM | single-company | |
arbeitsagentur |
Arbeitsagentur | board-wide | German federal job board |
arbeitnow |
Arbeitnow | board-wide | |
remoteok |
RemoteOK | board-wide | |
remotive |
Remotive | board-wide | |
weworkremotely |
We Work Remotely | board-wide | |
workingnomads |
Working Nomads | board-wide | |
nodesk |
NoDesk | board-wide | |
himalayas |
Himalayas | board-wide | |
jobicy |
Jobicy | board-wide | |
jobspresso |
Jobspresso | board-wide | |
4dayweek |
4 Day Week | board-wide | |
justjoin |
JustJoin.it | board-wide | Polish IT board |
nofluffjobs |
NoFluffJobs | board-wide | Polish IT board |
solidjobs |
SolidJobs | board-wide | Polish IT board |
thehub |
The Hub | board-wide | Nordic startups |
landingjobs |
Landing.jobs | board-wide | |
getonbrd |
Get on Board | board-wide | LatAm |
glints |
Glints | board-wide | SE Asia |
jobstreet |
Jobstreet / SEEK | board-wide | SE Asia / ANZ / HK — SEEK v5 JobSearch API (/api/jobsearch/v5/search). One adapter, many markets: pick with siteKey — ID-Main (id.jobstreet.com), SG-Main, MY-Main, and HK-Main (hk.jobsdb.com — Hong Kong keeps the JobsDB brand on the same SEEK platform). Migrated off the retired chalice-search v4 endpoint — v1.227.5 |
themuse |
The Muse | board-wide | |
jobbankca |
Job Bank (Canada) | board-wide | jobbank.gc.ca public ATOM feed (Canada's federal national employment service); host-pinned SSRF guard, config-driven keywords with a profile-target fallback — v1.212.0 |
mycareersfuture |
MyCareersFuture | board-wide | api.mycareersfuture.gov.sg public search API (Singapore's national job bank); POST + query-string pagination, host-pinned, config-driven keywords with a profile-target fallback — v1.213.0 |
larajobs |
LaraJobs | board-wide | RSS — v1.118.0 |
senjob |
Senjob | board-wide | senjob.com listing (Senegal — the scanner's first African source), HTML-parsed in-process, host-pinned; throws on a broken listing page rather than reporting an empty board — v1.210.0 |
yourator |
Yourator | board-wide | yourator.co/api/v4/jobs JSON API (Taiwan tech/digital); walks every page; employer ATS link emitted (display-only), API host-pinned — v1.211.0 |
torre |
Torre | board-wide | search.torre.co/opportunities/_search POST/JSON opportunity search (pan-LatAm remote roles, torre.ai); host-pinned HTTPS + redirect:error, single capped 20-row request (the API cannot page), configure search (+ experience) and optional remote_only — v1.221.0 |
itviec |
ITviec | board-wide | itviec.com HTML listing scrape (Vietnamese tech board); host-pinned HTTPS + per-company fail-soft; configure search/location — v1.227.5 |
careerviet |
CareerViet | board-wide | careerviet.vn HTML listing scrape (broad Vietnamese board); host-pinned HTTPS + page caps — v1.227.5 |
telegram |
Telegram | board-wide | Public channel preview at t.me/s/<channel>, configured in a top-level telegram_channels: block (not tracked_companies — a channel is not an employer and has no careers URL to detect). Host pinned to t.me exactly, HTTPS-only. Posts are PROSE, not job records: title = first substantive line, company/location only from explicit labels, unlabelled rows attributed to @channel rather than guessed; an empty parse throws instead of reading as "no vacancies". Lean on title_filter to drop ads and digests — v1.228.0 |
higheredjobs |
HigherEdJobs | board-wide | RSS category feeds — v1.117.0 |
hackernews |
Hacker News (Who is hiring) | board-wide | Monthly thread |
rss |
RSS | generic | Point it at any job RSS/Atom feed — the escape hatch when a board has no adapter |
Querying the results (v1.228.2).
GET /api/scan-results?q=matches search terms independently, combined with AND —менеджер продуктаalso finds "Менеджер по развитию продукта", and word order does not matter. It was a single whole-phraseincludes(q)until v1.228.2, which found only the exact adjacency:продакт менеджерreturned 32 rows where 162 carried both words. That is worth knowing when a search looks thin — job titles vary in wording far more than expected, so a short distinctive stem (продукт,product) beats a two-word phrase.
Telegram is the one source configured outside tracked_companies, because a channel
is not an employer — the scanner filters that list through detectApi(), and a
channel has no careers URL to detect. expandTelegramChannels() in
server/lib/en-scanner.mjs expands the block into ordinary adapter-selected entries,
so quarantine, filters and dedup all run on one scan path rather than a parallel one.
telegram_channels:
enabled: true
max_posts: 100 # default cap per channel (hard cap 300)
channels:
- { name: "PHP jobs", channel: rabotaphp }
- { name: "Salary PM", channel: salary_pm, max_posts: 50 }channel: accepts rabotaphp, @rabotaphp, https://t.me/rabotaphp, or a link to a
single post. Each channel is read from its public web preview at https://t.me/s/<channel> —
plain server-rendered HTML, no bot token and no API key. (Telegram publishes no RSS, and the
Bot API cannot read a channel your bot does not administer, so neither obvious route works.)
Three behaviours are deliberate and should not be "fixed":
-
A channel post is prose, not a job record. No structured fields exist. Separating real
postings from ads and digests is your
title_filter's job — the same one already tuned for other sources. These are leads, not listings. -
The company is never guessed. With no explicit label the row is attributed to the
channel (
@rabotaphp); a wrong employer would enter the tracker as fact. -
An empty parse is an error.
t.meanswers a private or missing channel with a redirect; returning zero rows would read as a quiet day and hide a typo in the config forever.
Parsing anchors on the three things a Telegram redesign is least likely to move:
data-post="<channel>/<id>" (which both windows the page and carries the post's only stable
URL), the message-text container, and the ISO <time datetime>.
Dispatched by ru-scanner.mjs::RU_DISPATCH and configured under russian_portals: in portals.yml (sources: ["hh", "habr", "trudvsem", "getmatch", "geekjob"] is the default).
| id | Label | configKey | Notes |
|---|---|---|---|
hh.ru |
hh.ru | hh |
HTML scrape of hh.ru/search/vacancy (the JSON API 403s programmatic clients) |
habr-career |
Habr Career | habr |
career.habr.com/api/... |
trudvsem |
Trudvsem | trudvsem |
Russian federal job board |
getmatch |
GetMatch | getmatch |
|
geekjob |
GeekJob | geekjob |
The full walkthrough lives in the in-app help guide §17 ("How to add a new job-portal source"). Summary:
-
Create
server/lib/sources/<slug>.mjsexportingmeta = { value, label, region: 'en'|'ru', configKey? }and asearch<Name>(query, opts)returning normalized job objects:{ id, title, company, url, salary, location, isRemote, workplaceType, relocates, date, snippet, source }. It auto-registers at boot — no registry edit. -
EN boards: add
server/lib/portals/adapters/<slug>.mjsimplementing the adapter contract —matches(url)andbuildEndpoint(...)(must return a string or null, never an object) — and register it inserver/lib/portals/registry.mjs::ALL_ADAPTERS. Update the count assertions intests/adapter-registry.test.mjsandtests/scan-sources-endpoint.test.mjs. -
RU sources: add a
RU_DISPATCHrow inru-scanner.mjs. -
SSRF host-pinning (mandatory): every adapter pins its hostname with an anchored regex (e.g.
/(^|\.)justjoin\.it$/i— sojustjoin.it.evil.comcan't spoof) and rejectshttp:. Adapters with no stable vendor host (e.g. Radancy) validate the endpoint path shape and are selected only via an explicitprovider:key inportals.yml. Parity tests must cover: evil host rejected,http:rejected, suffix-spoof rejected, parse happy path, pagination stop condition. -
Config-driven providers: POST/GraphQL/paginated boards thread
opts.companythroughbuildEndpoint; keepbuildEndpointreturning a string. - Add a parity test suite (
tests/sources-parity-*.test.mjs, stubbed transport — no network), bump the registry counts, and fan out docs ×17 per the Release Process.
- Entry point:
GET /api/stream/scan?source=ats|regional|both(SSE, onedoneper phase,final:trueon the last). -
dryRun=1skips writes todata/scan-history.tsv/data/last-scan.json;company=<slug>narrows the ATS walk. - Results are deduped against
data/scan-history.tsv, sanitized (scan-sanitize.mjs), quarantine-filtered (scan-quarantine.mjs), location-filtered, and matched against your target roles / two-pager (◎badge). - Client disconnect aborts in-flight upstream fetches (
AbortSignal).
Since v1.119.1 the #/scan Source dropdown's offline fallback (FALLBACK_SOURCES in public/js/views/scan.js) is gated by tests/scan-fallback-sources.test.mjs — exact value+label parity with the source registry, so a new adapter can't ship without the client list.