Rebuilding a competitor-monitoring crawler
The problem
A job tracking ~3,000 competitor browser extensions was failing in several ways at once.
It crawled serially — 30+ minutes a run. It had no concurrency guard, so two overlapping
runs shared one temp table that either could TRUNCATE, destroying the other's work.
Persistence was all-or-nothing, so a run dying at 80% discarded everything it had learned.
A single blank field from a single extension cancelled the whole run. And it had quietly
stopped running weeks earlier without anyone noticing.
What I built
- A pure classification engine deciding what each outcome means — first-seen, version-changed, delisted, relisted, blocked. Zero I/O, exhaustively unit-tested.
- Batched concurrency against the source, taking the same workload from 30+ minutes to minutes.
- Redis distributed locking (
SET NX EX) with TTL-backed release, so overlapping runs are impossible and a crashed process can't wedge the schedule shut. - Per-batch persistence — a run that dies partway keeps what it learned.
- Three entrypoints: an unattended cron, an authenticated internal API, and an operator CLI with
--planand--dry-runthat refuses a full live run without an explicit flag. - Noise suppression — ~2,600 catalogue entries are long-dead. Reporting them all would bury the dozen real changes, so they're counted but not itemised.
The bug underneath
While building it I found the ratings column was declared numeric(100,0).
Scale zero means Postgres rounds on write — a 4.6-star product had been stored as
5 for years. Invisible the whole time, because nothing had ever compared
the values. Fixed with a reversible migration.
- Node.js
- TypeScript
- PostgreSQL
- Redis
- Playwright
- Zod