Question about the KafkaNodePool resources ("storage overrides")

[BBQigniter]

In the changelogs of Strimzi v0.43 following is mentioned:

The storage overrides for configuring per-broker storage class are deprecated and will be removed in the future.
If you are using the storage overrides, you should migrate to KafkaNodePool resources and use multiple node pools with a different storage class each.

Does this mean that something like this in a Kafka-cluster manifest, will not work anymore some day in future?

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: kafka-test
  namespace: kafka-test
spec:
  kafka:
    version: 3.7.0
    replicas: 3
...
    storage:
      type: persistent-claim
      # has no effect but we need it
      size: 5Gi
      class: local-kafka-storage
      deleteClaim: false
    metricsConfig:
      type: jmxPrometheusExporter
...

In our case we just prepare mount-points/data-directories and define persistent volumes for example on three different Kubernetes nodes, where the Kafka/Zookeeper node pods then will be scheduled. So kafka-test-kafka-0 is always running on node-1, kafka-test-kafka-1 is always running on node-2 and so on. This makes setups quite easy and we do not have/need a shared or clustered filesystem that can be used by all nodes.

[scholzj]

Does this mean that something like this in a Kafka-cluster manifest, will not work anymore some day in future?

Yes and no 🫣. The YAML you shared will stop working. In order to use KRaft (ZooKeeper-less Kafka) you will need to move to KafkaNodePools and KafkaNodePool configure the storage in the KafkaNodePool resource.

However, the deprecated part that will not be supported anymore with KafkaNodePools is about the use of the storage class overrides:

    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
        class: my-storage-class
        overrides:
        - broker: 0
          class: my-storage-class-zone-1a
        - broker: 1
          class: my-storage-class-zone-1b
        - broker: 2
          class: my-storage-class-zone-1c

You do not seem to be using those. So you should be able to easily migrate to the KafkaNodePools as described in the docs.

[BBQigniter]

hmmhmmhmm - not extremely fond of this, but well. Thanks for the explanation 😊

With which version is it planned to completely drop the old method?

We have to re-setup our Kubernetes clusters anyway in the next few months and then I would just completely redo the Strimzi-setup too.

[scholzj]

hmmhmmhmm - not extremely fond of this, but well. Thanks for the explanation 😊

With which version is it planned to completely drop the old method?

We have to re-setup our Kubernetes clusters anyway in the next few months and then I would just completely redo the Strimzi-setup too.

Not sure what part are you referring to here.

[BBQigniter]

We are currently using Strimzi 0.40 and I'm planning to upgrade to 0.44.

With which version is it planned to completely drop the old method?

I assume our current config still will work with 0.44 but someday it will not work at all anymore or do I have to migrate to "KafkaNodePools" before even upgrading to any newer version?

[scholzj]

Well, if you had a better idea of how to accomodate the Kafka KRaft architectures you could have commented when these things were proposed and dicusssed. Now it is a bit late for that.

ZooKeeper-based clusters will be supported until Strimzi 0.45. Latest within 0.45 you will need to migrate to node pools and to KRaft (before upgrading to Strimzi 0.46 or newer).

[BBQigniter]

Sorry, if my posts sounded offending or so - I absolutely didn't mean to 😔. I really appreciate your work and Strimzi, it made it a lot easier for us to be introduced into the world of Kafka 😄. I'm pretty sure there is a very good reason for the change, else you wouldn't have done it 😄 - I'm only not very fond of it, because it means I'll have to change stuff, test and try to understand what's going on with the new setup, which will take time that I currently do not really have with all the other tasks I have to do.

Thanks for the support and clarification when Zookeeper-based clusters will vanish.