Our Entire Event Bus Is Now a Single Postgres Table
Andika's AI AssistantPenulis
Our Entire Event Bus Is Now a Single Postgres Table
You've heard the story a thousand times. A growing application, a move to microservices, and the inevitable introduction of a dedicated message broker. Kafka, RabbitMQ, SQS—these are the titans of event-driven architecture, the "right" way to decouple services. We followed that path, but the operational overhead, hidden complexities, and constant tuning became a significant tax on our small team. That’s when we asked a radical question: what if we could do less? It turns out we could. In a move that simplified our stack immensely, our entire event bus is now a single Postgres table, and it’s more reliable than ever.
This isn't just a story about technical simplification; it's about challenging the default architectural choices that the industry often presents as gospel. By leveraging the transactional power we already had in our primary database, we eliminated an entire class of distributed systems problems and regained focus on what truly matters: building features.
The Allure and Agony of Dedicated Message Brokers
Dedicated event buses promise a world of perfect decoupling and infinite scalability. They act as a central nervous system for your application, allowing services to communicate asynchronously without direct knowledge of one another. This is the textbook definition of a robust, event-driven architecture.
However, this power comes at a cost—one that isn't always immediately apparent.
Operational Overhead: You now have another complex, stateful system to provision, monitor, patch, and secure. Is your Kafka cluster properly configured? Are your RabbitMQ queues backing up? This becomes a full-time job in itself.
Transactional Integrity Nightmares: The classic problem is the dual-write. Your service needs to save data to its own database and publish an event. What happens if the database commit succeeds, but the message publish fails? You're left with inconsistent state. Solutions like the exist, but they add yet another layer of complexity.
Created by Andika's AI Assistant
Full-stack developer passionate about building great user experiences. Writing about web development, React, and everything in between.
Elusive Delivery Guarantees: While brokers offer various delivery guarantees like "at-least-once" or "at-most-once," achieving true "exactly-once" semantics is notoriously difficult and often misunderstood. Misconfigurations can easily lead to duplicate or lost messages.
For many teams, the complexity of managing a dedicated broker far outweighs the benefits, especially when their message volume is in the thousands or tens of thousands per day, not millions per second.
The Simplicity of a Postgres-Based Event System
The core idea behind using a database as a message bus is that a database table is, at its heart, a durable, transactional, and queryable log. These are the fundamental properties you need for a reliable queue. Instead of pushing an event to an external broker, you simply INSERT a row into a dedicated table within the same database transaction as your primary business logic.
This single-table message queue approach immediately solves the dual-write problem. The business data and the event are committed together, atomically. It’s an all-or-nothing operation, guaranteed by the ACID properties of your relational database. This shift turns a distributed systems problem into a simple database transaction.
Implementing the Single-Table Message Queue
Putting this Postgres-based event system into practice is surprisingly straightforward. It revolves around a well-designed table and a clever locking mechanism that has been available in Postgres for years.
The Anatomy of the events Table
Everything starts with the table schema. A robust implementation looks something like this:
payload: A JSONB column to store the event data. It's flexible and indexable.
topic: A string to categorize events, allowing different consumers to process different types of messages.
status: A state machine for the event (pending, processing, done, failed). This is crucial for tracking and debugging.
process_at: Allows for scheduling events to be processed in the future.
The Producer and Consumer Dance
With the table in place, the logic for producing and consuming events is simple.
Producing an Event:
A producer's job is to insert a row into the event_queue table. The magic is doing this inside the same database transaction as the state change that generated the event.
BEGIN;-- 1. Perform the business logicUPDATE orders SETstatus='confirmed'WHERE id =123;-- 2. Enqueue the corresponding eventINSERTINTO event_queue (topic, payload)VALUES('order.confirmed','{"order_id": 123, "customer_id": 456}');COMMIT;
If either the UPDATE or the INSERT fails, the entire transaction is rolled back. Your system state remains perfectly consistent.
Consuming an Event:
Consumers are worker processes that poll the table for pending jobs. To prevent multiple consumers from grabbing the same event, we use a powerful Postgres feature: SELECT ... FOR UPDATE SKIP LOCKED.
-- This query atomically finds and locks the next available eventWITH next_event AS(SELECT id
FROM event_queue
WHEREstatus='pending'AND process_at <=NOW()ORDERBY process_at
LIMIT1FORUPDATE SKIP LOCKED
)UPDATE event_queue
SETstatus='processing', attempt_count = attempt_count +1WHERE id =(SELECT id FROM next_event)RETURNING*;
This query is the heart of the consumer. Here’s what it does:
It looks for the oldest pending event.
FOR UPDATE places a lock on the selected row, making it invisible to other transactions.
SKIP LOCKED tells other concurrent consumers to ignore the locked row and try to find the next available one.
The UPDATE statement marks the job as processing and returns it to the worker.
This pattern provides at-least-once delivery and allows you to build a pool of competing consumers that can process events in parallel safely and efficiently.
Key Benefits and Trade-offs
Adopting a single Postgres table for our event bus brought significant wins, but it's not a silver bullet. It’s essential to understand the trade-offs.
The Wins:
True Transactional Integrity: Events are created atomically with business data. No more data consistency bugs.
Operational Simplicity: We manage one less piece of infrastructure. Our monitoring, backup, and security procedures are unified.
Simplified Tooling and Debugging: An event is just a database row. We can use standard SQL to inspect the queue, retry failed jobs, or perform analysis.
Lower Costs: We eliminated the hosting and maintenance costs associated with a separate message broker cluster.
The Trade-offs:
Limited Throughput: Postgres is not Kafka. This approach is not suitable for systems that need to process hundreds of thousands of events per second.
Potential Database Contention: High-frequency polling and processing can put extra load on your primary database, potentially impacting application performance.
Polling Inefficiency: Constant polling can be less efficient than the push-based models of dedicated brokers.
Basic Feature Set: You don't get advanced features like complex routing topologies or stream processing out of the box.
Is a Postgres Event Bus Right for You?
This approach isn't for everyone. The decision depends entirely on your scale, team size, and specific requirements.
Consider a Postgres-based event system if:
You have a low-to-moderate event volume (e.g., up to a few hundred events per second).
Transactional atomicity is your highest priority.
You want to simplify your tech stack and reduce operational overhead.
Your team is already proficient with PostgreSQL.
Stick with a dedicated broker like Kafka or RabbitMQ if:
You need to handle very high-throughput event streaming (e.g., IoT, clickstream data).
You require complex event routing, fan-out, or stream processing capabilities.
Your event queue load is so high it would negatively impact your primary database's performance.
Conclusion: Simplicity Is a Feature
Our journey to a single-table event bus was a lesson in pragmatism. We traded the "theoretically perfect" architecture for one that was "practically better" for our specific context. By leveraging the powerful, transactional foundation of PostgreSQL, we built a simpler, more reliable, and more maintainable system.
Before reaching for another piece of infrastructure, look at the tools you already have. Sometimes, the most elegant solution is the one that's been right in front of you all along.
Have you tried using your database as a message queue? Share your experiences or challenges in the comments below. Let's challenge the default architectures together.
