We Killed Our Sidekiq Cluster with Postgres LISTEN/NOTIFY

We Killed Our Sidekiq Cluster with Postgres LISTEN/NOTIFY

You've been there. A critical system, humming along for months, suddenly starts to choke. Job latencies skyrocket, memory usage balloons, and inexplicable errors flood your monitoring dashboards. You blame recent deploys, network hiccups, or a sudden traffic spike. For us, the culprit was far more insidious: a clever architectural pattern that backfired spectacularly. In our quest for a real-time, polling-free system, we killed our Sidekiq cluster with Postgres LISTEN/NOTIFY, and this is the story of how we did it—and how you can avoid our fate.

This post-mortem is for any engineering team tempted by the elegance of database-driven events. While powerful, this pattern hides a deadly trap for connection-pooled applications like Sidekiq, and understanding the mechanics of this failure is crucial for building resilient systems.

The Allure of Real-Time: Why We Chose Postgres LISTEN/NOTIFY

Our goal was simple: when a specific record changed in our database, we needed to trigger a background job immediately. The standard approach? Polling. A cron job or a scheduled Sidekiq job would wake up every few seconds, query the database for changes, and enqueue jobs accordingly.

But polling feels inefficient. It’s a constant, low-level hum of activity that generates database load, even when there’s nothing to do. We wanted something more elegant, more event-driven. Enter PostgreSQL's LISTEN/NOTIFY functionality.

It’s a beautiful concept: a lightweight, asynchronous messaging system built directly into the database.

1. NOTIFY: You can send a notification on a named "channel" from anywhere, typically from a database trigger or function.
2. LISTEN: A client can issue a LISTEN command on that channel. The client's connection will then block until a notification arrives.