Why Your Cron Jobs Run Three Times: Zookeeper Leader Election for Horizontally Scaled Node.js

A scheduled broadcast that runs inside your application process does exactly what you told it to: it fires on schedule, on every instance. Run three instances behind a load balancer and real people receive the same message three times. Nothing crashed. No error was logged. Every instance behaved correctly — and the system, as a whole, misbehaved in the most public way a chatbot platform can.

This is the problem we hit scaling Hexabot, an open-source chatbot builder (NestJS + Next.js, 700+ GitHub stars) adopted by major Tunisian telecom providers and serving 500+ daily active users. Stateless request handling scales horizontally for free. Cron jobs do not.

The Horizontal Scaling Problem

Several background tasks — message cleanup, analytics aggregation, session expiry, scheduled broadcasts — ran on schedules inside the application process. The moment you run N instances, every job runs N times:

The duplicated broadcast is the loud failure. The quiet ones are worse: cleanups racing each other and analytics counted N times corrupt data silently, and you find out weeks later in a report that does not add up.

The Naive Fixes, and Why They Fail

"Just designate a cron instance." Run schedulers on one special node, disabled everywhere else. Now you have a snowflake deployment: a single point of failure that takes every background job down with it, a config flag that someone will eventually set wrong, and a rolling deploy window where the cron node briefly exists twice — duplicating jobs anyway.

"Just grab a database lock." Each instance tries to acquire a lock row before running a job. Closer, but now you own the hard parts yourself: locks leaking when an instance dies mid-job, clock skew between instances deciding "who is first," and per-job lock bookkeeping that grows with every new task. You have started writing a distributed coordination system by accident — the category of software with the highest bug-per-line density there is.

The Solution: Zookeeper Leader Election

Coordination is a solved problem — the right move is to delegate it to a system built for nothing else. We used Apache Zookeeper's classic leader election recipe:

In application code, the check is one guard at the top of every scheduled task:

// Only the leader instance runs cron jobs
const isLeader = await zookeeperClient.isLeader();
if (!isLeader) return;
 
await runScheduledJob();

Two properties of Zookeeper do all the heavy lifting that the naive fixes fumble:

• Ephemeral znodes are liveness. The registration exists only while the instance's session is alive. A crashed leader does not leak a lock — its znode vanishes with it, and the next candidate is promoted automatically, usually within seconds.
• Sequence numbers are consensus. "Lowest number wins" is unambiguous on every instance without any instance talking to another. There is no clock skew to argue about and no split-brain window where two nodes both believe they are leader.

Every instance stays identical — same image, same config, no snowflakes — and any of them can become the leader. Horizontal scaling stays boring, which is the point.

A footnote from the same scaling push: the Docker build was taking 8+ minutes in CI, and multi-stage builds, dependency-layer caching, and a proper .dockerignore cut it to under 4 (a 53% reduction) — worth doing, but it is plumbing, not the story.

What I Learned

Do not hand-roll coordination. Leader election, distributed locks, and consensus look simple in the happy path and fail in the crash path. Delegate to a system whose entire job is the crash path.

Zookeeper is a real operational dependency. It buys correctness, but you now run and monitor a coordination service (or pay for a managed one). For a single-team app with one or two background jobs, a database advisory lock — eyes open to its failure modes — can be the honest choice; we crossed the line where owning those failure modes cost more than owning Zookeeper.

Backwards compatibility is a constraint, not a preference. When telecom companies depend on your API, "we cleaned up the interface" is an outage with release notes. Open-source adoption converts every public surface into a promise.

Related Reading

• Building Event-Driven Systems at Scale — async pipelines and queue-based decoupling in a production hiring platform