Kafka+SQL improve bootstrapping by one-time replication of state between replicas?

[EricWittmann]

A potentially significant problem with the Kafka+SQL storage implementation is the bootstrapping time for a replica. Each replica, when it starts up, must fully consume all messages in the Kafka topic. We are using log compaction to minimize the number of messages that may exist, but even so as the size of the registry grows, so too will the number of messages on the topic.

We should explore the idea of pulling the current state of the in-memory database directly from a peer replica (if one exists and can be found). It's possible that we could perform a database-specific dump and send it over the wire to the new replica in less time than it would take to consume all of the Kafka messages.

There is a question of how to then skip the appropriate number of Kafka messages so that we sync back up with normal message consumption.