LinkedIn today announced Northguard, a scalable log storage system that replaces Kafka, and Xinfra, a virtualized Pub/Sub layer. Northguard delivers sharded data & metadata, log striping, strong consistency, and self-balancing clusters at a larger scale than Kafka, while Xinfra enables seamless migration and unified access across Kafka and Northguard. According to LinkedIns engineers, Kafka had become increasingly difficult to manage at LinkedIns scale. Northguards architecture eliminates Kafkas single-controller and partition-based limitations. Northguards data model organizes logs into records, segments, ranges, and topics. This fine-grained structure enables balanced load, high availability, and seamless scaling. Brokers naturally self-balance as producers produce new segments. New segments get added to the range and get assigned to potentially new brokers. Compared to traditional indexed partitions, ranges provide a more flexible scaling mechanism. Range splits only interrupt clients writing to the affected range while maintaining total ordering guarantees. Northguards metadata model uses sharded, Raft-backed replicated state machines, distributed across vnodes. LinkedIn optimized Northguards protocols for performance and durability. Metadata operations such as create, delete, and query use unary request/response calls routed to vnode leaders. Produce, consume, and replication flows are sessionized streaming protocols with pipelin
