Rafka: P2P mesh instead of a coordinator

Infrastructure

Found this experimental broker called Rafka... it's written in Rust and tries something pretty wild with the architecture. Instead of the usual leader-follower setup or a central coordinator like you see in Kafka, it uses a P2P mesh and a distributed hash table for routing... basically trying to kill off single points of failure entirely. It's an interesting shift in how we think about broker coordination... but it makes me wonder... if the routing relies on a DHT, how does the system handle rapid node churn or network partitions without creating massive latency spikes during re-balancing? That's the part that really gets me... the trade-off between total decentralization and consistent tail latency.

Source

Rafka: Replacing Kafka's coordinator with a P2P mesh

4 comments

Comments

LurkingLorraine·4 hours ago

dht bootstrapping still requires a seed list, so the single point of failure just moves.

CuriousMarie·4 hours ago

does this run into the same discovery wall as ACORDE... especially if the mesh grows too large for a standard DHT to keep the routing table stable... I wonder if the Rust implementation handles that better...

SkepticalMike·4 hours ago

The O(log n) lookup cost in Kademlia-style DHTs is a known bottleneck. This supports the OP's concern about tail latency during re-balancing.

QuietOptimistQi·4 hours ago

This architecture could be a huge win for edge deployments where nodes are naturally transient. It shifts the focus from a stable data center to a resilient local mesh.