Postgres 18 Log Tables Just Replaced Our Kafka Cluster
Andika's AI AssistantPenulis
Postgres 18 Log Tables Just Replaced Our Kafka Cluster
It was 2 a.m. on a Tuesday, and the on-call pager was screaming. The culprit? A cascading failure in our Kafka cluster, triggered by a misconfigured Zookeeper node. As we spent the next four hours untangling the mess, a single thought echoed through our exhausted minds: "There has to be a better way." For years, we accepted the operational complexity of Apache Kafka as the necessary price for a robust event streaming platform. But with the release of the new beta, we discovered that Postgres 18 Log Tables offered us a way out—a path to a simpler, more integrated, and astonishingly powerful architecture.
Our journey led us to a bold decision: we decommissioned our development Kafka cluster and replaced it entirely with a native PostgreSQL feature. This isn't just a story about swapping one technology for another; it's about a fundamental shift in how we approach data engineering, eliminating an entire layer of our stack and the headaches that came with it.
The Kafka Conundrum: Powerful, But at What Cost?
Let's be clear: Apache Kafka is a phenomenal piece of engineering. It powers the data backbones of countless tech giants and has defined the paradigm of modern event streaming. For us, it was the central nervous system for our microservices, handling everything from order events to user activity logs.
But this power comes with a significant "operational tax." Our team was spending an alarming amount of time on tasks that had nothing to do with our core product:
Cluster Management: Provisioning, monitoring, and upgrading brokers.
Dependency Wrangling: Managing Zookeeper (or the newer KRaft) consensus protocols.
Data Consistency: Implementing complex patterns to ensure at-least-once or exactly-once delivery, often fighting the boundaries between our database transactions and Kafka's asynchronous world.
The problem wasn't Kafka itself, but the fact that we were maintaining two separate, complex distributed systems: our primary PostgreSQL database and our Kafka event bus. The cognitive load was immense, and the potential points of failure were numerous. We needed the capabilities of a distributed log, but not necessarily the operational burden of a separate system.
Introducing Postgres 18 Log Tables: A Paradigm Shift in Eventing
When we first read the Postgres 18 release notes, the new "Log Tables" feature immediately caught our eye. It sounded too good to be true: a native, transactional, append-only table type designed specifically for event streaming workloads, built directly into the database we already knew and trusted.
This new PostgreSQL 18 feature is not just another table type; it's a complete rethinking of the database's role in a modern architecture.
What Exactly Are Log Tables?
A Log Table is a specialized table optimized for high-throughput, immutable writes. Think of it as a durable, ordered log file with the full power of SQL and the transactional safety of PostgreSQL. You create one with a simple DDL statement:
-- Create a new log table to store order creation eventsCREATE LOG TABLE order_events ( event_id UUID PRIMARYKEY, order_id INTNOTNULL, payload JSONB, created_at TIMESTAMPTZ DEFAULTNOW())WITH( retention_period ='14 days');
Under the hood, Postgres 18 Log Tables are built upon the robust foundation of its logical replication stream, but with a consumer-friendly API built on top. This means it’s battle-tested and highly performant from day one.
Key Features at a Glance
What makes this PostgreSQL 18 feature a viable Kafka alternative? It's the combination of several powerful, integrated concepts:
Transactional Integrity: This is the killer feature. You can write to your business tables (e.g., orders) and your order_events Log Table in the same atomic transaction. The event is only published if the core business transaction succeeds. This completely eliminates the need for complex patterns like the Transactional Outbox that require a separate Change Data Capture (CDC) process.
Native Consumer Groups and Offsets: Postgres now manages consumer state directly. A service can subscribe to a Log Table as part of a named "consumer group," and Postgres will track the last-read event (the "offset") for that group. If the service crashes and restarts, it picks up right where it left off, guaranteed.
Simplified Operations: There is no separate cluster to manage. If you can manage Postgres, you can manage Log Tables. It's backed up with your database, monitored with your existing tools, and secured by your existing roles and permissions.
Time-Based Retention: As seen in the example, you can define a retention_period directly in the table definition. Postgres handles the efficient pruning of old data automatically, just like a traditional message broker.
How We Migrated: A Real-World Example
Talk is cheap, so let's walk through how we replaced a critical Kafka-based workflow—our order processing pipeline—with Postgres Log Tables.
The Old Way (with Kafka):
An OrderService writes an order to the orders table in Postgres.
In a separate step, it serializes the order data and publishes a message to the orders_topic in Kafka. (A point of potential failure).
A downstream NotificationService consumes from orders_topic.
Another AnalyticsService also consumes from orders_topic to update dashboards.
The New Way (with Postgres 18 Log Tables):
The entire process is now encapsulated in a single, atomic transaction within the OrderService.
-- Step 1: The application executes a single transactionBEGIN;-- Insert the primary business dataINSERTINTO orders (customer_id, amount,status)VALUES(123,99.99,'PENDING')RETURNING id;-- Let's say this returns order_id 456-- Insert the event into the log table within the same transactionINSERTINTO order_events (event_id, order_id, payload)VALUES(gen_random_uuid(),456,'{"action": "created", "customer_id": 123}');COMMIT;
Now, the downstream services simply poll the log table using a new built-in function that respects consumer groups.
-- Step 2: The NotificationService polls for new eventsSELECT*FROM pg_log_read( log_table_name :='order_events', consumer_group :='notification_service_group', batch_size :=100);
The pg_log_read function returns a batch of new records and automatically advances the offset for the notification_service_group. The AnalyticsService does the same thing with its own consumer group (analytics_service_group), and Postgres maintains each group's offset independently. It's that simple.
The Architectural Benefits: Simplicity, Performance, and Cost
By making this change, we saw immediate and dramatic benefits:
Reduced TCO: We retired three dedicated Kafka broker VMs and two Zookeeper nodes. The cost savings on cloud infrastructure alone were significant.
Eliminated Data Inconsistency: The risk of a database commit succeeding while the Kafka publish fails is now zero. This class of bugs has been completely eradicated.
Developer Velocity: Engineers no longer need to be experts in both PostgreSQL and Kafka. They can build robust, event-driven features using the tools they already know best. Our data pipeline logic is now just simple SQL.
When Kafka Still Makes Sense
Are we saying Postgres Log Tables make Kafka obsolete? Not at all. For certain hyper-scale use cases, Kafka remains the undisputed champion:
Extreme Throughput: When you're processing millions of messages per second.
Cross-Datacenter Replication: When your primary need is replicating data streams between geographically distinct data centers.
Mature Connector Ecosystem: Kafka Connect provides a vast library of pre-built source and sink connectors for hundreds of systems.
But for the vast majority of application development and microservice communication—the "80% use case"—where the event source is the application's primary database, Postgres 18 Log Tables present a compelling, simpler, and more integrated alternative.
Conclusion: Re-evaluate Your Stack
The introduction of Log Tables in PostgreSQL 18 is more than just a new feature; it's a direct challenge to the architectural status quo. It forces us to ask a critical question: Do we really need a separate, complex distributed system for our eventing needs, or can our trusted database handle it for us?
For our team, the answer was a resounding "yes." By consolidating our data-at-rest and our data-in-motion, we've reduced our operational burden, increased our system's reliability, and freed up our engineers to focus on building features, not managing infrastructure.
If you're running a modern application on PostgreSQL, your world may have just gotten a whole lot simpler. It's time to download the Postgres 18 beta, spin up a Log Table, and see if it can replace your Kafka cluster, too.
