Discovery: Connectors, Quarantine & Dedup
Six job sources, one typed interface, and a pipeline that treats every external item as guilty until validated.
Every job source — Greenhouse, Lever, GitHub lists, Apify scrapers, Instagram captions — is an adapter with two contracts: what my config looks like and what one raw item must look like. Items that fail validation go to quarantine with their raw JSON preserved, never crash the batch. Postings get a fingerprint so the same job seen on three sites becomes one row with three receipts — and freshness data, not duplicates.
Fresh postings are the product's oxygen — the spec's rule is "freshness beats breadth." But scraped data is the least trustworthy input a system can have: schemas drift, fields vanish, HTML changes shape overnight. Wera's discovery layer is built like a harbor receiving foreign cargo: every source ship gets a sworn translator (the connector), every crate is opened at the dock (schema validation), suspicious crates go to a bonded warehouse with their seals intact (quarantine), and repeat shipments of the same goods are recognized by their manifest, not their packaging (the fingerprint).
Follow one nightly run
The build journey
Define the connector contract
A connector is five things: a key, a provider, two zod schemas, and run/normalize functions. Nothing else about a source is allowed to matter.
SourceConnector<TInput, TRawItem> — inputSchema validates config before a run; rawItemSchema validates each external item at ingestion.
The spec lists "output schemas can change" as a top risk. Two explicit contracts turn upstream drift into a measurable event instead of a mystery crash.
Six implementations share the shape: Greenhouse (board API), Lever (postings API), public-list (parses JSON, HTML tables, and Markdown pipe tables from GitHub READMEs), a generic Apify actor connector with LinkedIn/Indeed presets, and Instagram — whose raw items are AI-extracted roles from captions.
Quarantine instead of crash, quarantine instead of drop
One drifted item marks itself parse_failed and the batch continues; the raw JSON is preserved for re-ingestion after a schema fix.
The whole policy is nine lines in ingestConnectorItems:
const parsed = connector.rawItemSchema.safeParse(raw);
if (!parsed.success) {
items.push({
status: "parse_failed",
raw,
reason: parsed.error.issues.map((issue) => issue.message).join("; "),
});
continue;
}
Ask the schema: does this raw item look the way this source promised? (safeParse answers instead of exploding.)
If it doesn't…
…file it as a quarantined item:
marked failed,
with the original untouched payload kept as evidence,
and every specific complaint the schema had, joined into one readable reason.
Then move on to the next item — one bad crate never sinks the ship.
Quarantined counts surface on the Sources screen per run — and the spec insists a run with failures never shows a green check. Schema drift becomes a dashboard number trending up, catchable on a Tuesday afternoon instead of a production alert at 6 a.m.
Fingerprint postings; badge disagreements, never merge them
SHA-256 of normalized company + title + location + season. A match means "same job, seen again" — freshness signal, not duplicate row.
createPostingFingerprint — normalization lowercases; strips seasons ("Summer 2027"), filler ("Intern", "co-op"), punctuation, and corporate suffixes ("Inc", "LLC"); canonicalizes "autumn"→"fall".
The same internship appears on LinkedIn, a GitHub list, and the company's own board — with cosmetically different titles. Multi-source appearance is itself information: the posting is real and active.
On match: update last_seen_at, clear probably_closed_at, append a source_signals row. If the apply URL or track disagrees, set a "possible duplicate" badge instead of merging — the spec's bias: a false-negative shows a badge; a false-positive hides a real job.
Schedule it, cap it, make retries free
A daily cron fans out over saved sources with a 500-item cap and a per-day idempotency key — running it twice is a no-op.
scheduled-discovery (07:00 Africa/Nairobi) in apps/web/src/trigger/discovery.ts, plus a manual "Run discovery" action capped at 200 items for responsiveness.
A curated GitHub list can hold 4,000 rows; unbounded ingestion blew serverless timeouts (a real fixed bug, commit fff6ddf). Caps make worst-case runtime a known number.
Run identity is sourceRun:scheduled:{connectorKey}:{isoDay} against a unique index — a platform retry of the same day's run upserts instead of double-ingesting. Progress counters flush every 50 items so a long run never looks like 0/0/0.
The central idea: adapters over integrations
A world traveler doesn't rewire her laptop for every country — she carries plug adapters. The wall socket's shape changes; the laptop never knows. Wera's connectors are plug adapters for job sources: each one knows its socket (Greenhouse's API, a GitHub README's table format, Instagram's caption chaos) and converts it to the one plug shape the rest of the system accepts (NormalizedSourceItem). Adding a seventh source means building one adapter — the pipeline, inbox, dedup, and health screens come free.
This is the adapter pattern under a pipes-and-filters pipeline, with validation contracts at the seam. The generic algorithm (ingestConnectorItems) is written once against the interface: parse config → run → per-item safeParse → normalize → fingerprint. Connectors carry no persistence logic; the store carries no source knowledge — the seam is a plain data structure (ConnectorIngestionResult), which is why the same result type flows from a manual button, a cron, or a test fixture.
The tradeoff is the least-common-denominator problem: the normalized shape can only hold what all sources can express, so source-specific richness (Greenhouse departments, Instagram engagement) survives only in the raw envelope. And note the most instructive connector: Instagram's "raw items" are AI-extracted roles — the extractor runs inside run() so extraction failures are per-post error values, while dataset-shape drift is deliberately passed through to fail rawItemSchema into visible quarantine. Even the failure routing is designed.
Apply it
Your university launches a jobs RSS feed. What does adding it to Wera require?
Greenhouse renames a field and 300 items/day quarantine for a week before anyone updates the schema. After the fix ships, what's recoverable?
"SWE Intern" and "Software Engineering Intern" at the same company/location/season produce different fingerprints and appear as two inbox rows. Is this a bug?
Ingestion pipelines & graceful degradation
Every serious system eventually ingests someone else's data, and someone else's data is always worse than promised. The transferable kit: validate per item, not per batch; quarantine with evidence instead of dropping; make partial success a first-class status; and derive identity from content (fingerprints) when upstream IDs can't be trusted across sources.
Best practices
- Two contracts per source: config schema and per-item schema — drift becomes measurable.
- Preserve raw payloads on failure; re-ingestion after a fix is the payoff.
- Cap batch sizes and timeout every fetch — worst-case runtime should be arithmetic, not hope.
- Choose your dedup error direction consciously, and tell the user with a badge instead of guessing silently.
Common pitfalls
- Batch-level try/catch: one malformed item aborts 499 good ones.
- Green dashboards on partial failure — if failures don't change the status color, nobody looks.
- Trusting upstream IDs as globally unique across sources.
- Normalizing so aggressively that distinct roles collide — collision direction is a product decision.
The canonical vocabulary for pipes-and-filters, dead-letter channels (quarantine's formal name), and message translators (connectors).
↗ Article Adapter — Refactoring.Gururefactoring.guruThe pattern behind SourceConnector, with diagrams — a ten-minute read that names what you just learned.
↗ Docs Apify documentationdocs.apify.comActors, runs, and datasets — the rented-scraper model Wera polls for its hardest sources.
↗