Blog post for moving data between JBOD disks using Cruise Control #469

ShubhamRwt · 2024-12-19T14:16:53Z

Type of change

Select the type of your PR

Typo/minor fix
New blog post (see the README for the process)
Other

Signed-off-by: ShubhamRwt <[email protected]>

ShubhamRwt · 2024-12-19T14:28:45Z

@scholzj @ppatierno Hi, is it just me or the formatting in the blog post is messed up for you also?

scholzj · 2024-12-19T14:30:39Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+If the data is not removed from the disk, and it is removed then potential data loss can happen.
+Currently, moving data between the JBOD disks is done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control
+
+## Cruise Control to move data between JBOD disks


If you move all your headers one level down, the webpage would look nicer ... i.e. ## -> ###, ### -> #### etc.

scholzj · 2024-12-19T14:31:02Z

@scholzj @ppatierno Hi, is it just me or the formatting in the blog post is messed up for you also?

Not sure what exactly you mean by it and where exactly.

ShubhamRwt · 2024-12-19T14:34:57Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+      volumeIds: [1, 2]
+```
+
+Now let’s wait for the `KafkaRebalance` resource to move to `ProposalReady` state. You can check the rebalance summary by running the following command once the proposal is ready:


@scholzj for eg. here. I see the text in green when it should be white as I have above for normal text

You mean in the GitHub review window? Or on the blog preview?

In the GitHub review window

So, from my experience, that is quite easily derailed by some code examples etc. So I would not make too muhc out of it as long as the preview looks good.

Okay, thanks for clearing the doubt. I though I have some wring formatting or something which is causing this

ShubhamRwt · 2024-12-19T14:35:40Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+
+### Additional notes
+
+1. This feature only works if JBOD storage is enabled


Here also I see the text in blue while these are just normal text and should be in white If if Iam not wrong?

scholzj · 2024-12-19T14:46:23Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+author: shubham_rawat
+---
+
+Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Kafka.


Kafka supports multiple disks. JBOD storage configuration type is more of a Strimzi thing. That is how we call it in the API. I would either reword it or change it from Kafka to Strimzi?

scholzj · 2024-12-19T14:47:06Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.
+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
+If the data is not removed from the disk, and it is removed then potential data loss can happen.


Maybe ...

Suggested change

If the data is not removed from the disk, and it is removed then potential data loss can happen.

If the data is not removed from the disk, and the disk is removed then potential data loss can happen.

Or you could adjust the following:

"In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
If the data is not removed from the disk, and it is removed then potential data loss can happen."

To:

"If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.
Failing to do so could result in data loss."

scholzj · 2024-12-19T14:47:37Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
+If the data is not removed from the disk, and it is removed then potential data loss can happen.
+Currently, moving data between the JBOD disks is done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control


Suggested change

Currently, moving data between the JBOD disks is done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control

Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control

As we're in 0.45 release, maybe change "we are introducing" to "we introduced"?
Also missing period at end of sentence.

scholzj · 2024-12-19T14:48:35Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+
+## Cruise Control to move data between JBOD disks
+
+This feature will allow you to move the data between the JBOD disks using the `KafkaRebalance` custom resource that we have in Strimzi.


Between the JBOd disks sounds not right here. That is what the intrabroker rebalancing does which we have for some time already. You are moving all data from one disk to another disks.

scholzj · 2024-12-19T14:48:52Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+## Cruise Control to move data between JBOD disks
+
+This feature will allow you to move the data between the JBOD disks using the `KafkaRebalance` custom resource that we have in Strimzi.
+This feature makes use of the `remove-disks` endpoint of Cruise Control that triggers a rebalancing operation which moves replicas, starting with the largest and proceeding to the smallest, to the remaining disks. 


Suggested change

This feature makes use of the `remove-disks` endpoint of Cruise Control that triggers a rebalancing operation which moves replicas, starting with the largest and proceeding to the smallest, to the remaining disks.

This feature makes use of the `remove-disks` endpoint of Cruise Control that triggers a rebalancing operation which moves all replicas, starting with the largest and proceeding to the smallest, to the remaining disks.

scholzj · 2024-12-19T14:50:10Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+spec:
+  replicas: 3
+  roles:
+    - broker


Should this be controller?

I guess that based on the rest of the YAMLs, @ShubhamRwt is making the example with a ZooKeeper based cluster but I totally agree on using KRaft only from now on.

Yes, I will update this with a KRaft cluster

scholzj · 2024-12-19T14:50:26Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+  zookeeper:
+    replicas: 3
+    storage:
+      type: persistent-claim
+      size: 100Gi
+      deleteClaim: false


Use KRaft please.

scholzj · 2024-12-19T14:52:27Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+    - brokerId: 0
+      volumeIds: [1, 2]


How do you know what will be broker 0? You should probably also clean it from all brokers from the node pool and take the example till the end to remove the disks?

@scholzj Sorry, I didn't understand what you meant by How do you know what will be broker 0. I was just showing how we can move all the data from the two volumes to some other volume

How do you know the ID 0 will not be a broker form the second node pool with only one volume? The YAML you use here does not give you deterministic order in which the node IDs will be assigned.

@scholzj Hi, before I push my changes if I understood the above suggestion correctly -> Iam now using a Kraft cluster with 3brokers and 3 controllers having 3 disks and then removing the 3rd disk from all the brokers(not the controllers) and taking the ecample till the end where we remove the disks

The point of this comment was mainly that without the node ID annotations, ou do not know if nodes 0, 1 and 2 will be brokers or controllers.

scholzj · 2024-12-19T14:53:44Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+
+### Additional notes
+
+1. This feature only works if JBOD storage is enabled


It only works when multiple disks are used, not when JBOD is enabled but has 1 disk, or?

scholzj · 2024-12-19T14:54:39Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+2. Make sure you have more than one volume per broker else you will be prompted of not having enough volumes to move the data to.
+3. This endpoint does not provide `before` load since upstream Cruise Control project does not support `verbose` with this endpoint so the `loadmap` generated should only have `afterLoad` information.
+
+## What's next


You should probably list the risks / incomplete parts as well ... in paritcular, new partition replicas might be scheduled to the disks between cleaning them up with cruise control and removing them which might lead to data loss again.

ppatierno · 2024-12-20T09:09:16Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+## Setting up the environment
+
+Let's set up a cluster to work through an example demonstrating this feature.
+To get the Kafka cluster up and running, we will first have to install the Strimzi Cluster Operator and then deploy the `Kafka` resource.


It's not going to be just Kafka but also KafkaNodePool.

ppatierno · 2024-12-20T09:10:20Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+metadata:
+  name: my-cluster
+  annotations:
+    strimzi.io/node-pools: enabled


missing annotation to specify it's a KRaft based cluster and so remove the ZooKeeper section as well.

ppatierno · 2024-12-20T09:11:04Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+spec:
+  replicas: 3
+  roles:
+    - broker


I guess that based on the rest of the YAMLs, @ShubhamRwt is making the example with a ZooKeeper based cluster but I totally agree on using KRaft only from now on.

PaulRMellor

Looking good, Shubham. I left a few suggestions. I thought the example might benefit from some kind of introduction to what we're trying to achieve and how.

PaulRMellor · 2025-01-07T15:28:43Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

@@ -0,0 +1,345 @@
+---
+layout: post
+title: "Moving data between the JBOD disks using Cruise Control"


Suggested change

title: "Moving data between the JBOD disks using Cruise Control"

title: "Moving data between JBOD disks using Cruise Control"

PaulRMellor · 2025-01-07T15:44:39Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.
+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
+If the data is not removed from the disk, and it is removed then potential data loss can happen.


Or you could adjust the following:

"In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
If the data is not removed from the disk, and it is removed then potential data loss can happen."

To:

"If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.
Failing to do so could result in data loss."

PaulRMellor · 2025-01-07T15:47:01Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+In case you plan to remove a disk, and it contains some partition replicas, then you need to make sure that data is safely moved to some other disks.
+If the data is not removed from the disk, and it is removed then potential data loss can happen.
+Currently, moving data between the JBOD disks is done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control


As we're in 0.45 release, maybe change "we are introducing" to "we introduced"?
Also missing period at end of sentence.

PaulRMellor · 2025-01-07T15:55:11Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+    segment.bytes: 1073741824
+```
+
+Once you create the topic, now you can check whether the volumes have some partition replicas assigned to them or not using the `kafka-log-dir.sh` tool. Let's see the partition replicas assigned to the volumes on broker with id 0.


Suggested change

Once you create the topic, now you can check whether the volumes have some partition replicas assigned to them or not using the `kafka-log-dir.sh` tool. Let's see the partition replicas assigned to the volumes on broker with id 0.

Now you can check whether the volumes have some partition replicas assigned to the topics using the `kafka-log-dir.sh` tool. Let's see the partition replicas assigned to the volumes on broker with id 0.

PaulRMellor · 2025-01-07T15:56:55Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+}
+```
+
+Now lets try to move the data of volume 1 and volume 2 to volume 0, present on broker with ID 0. For doing that let's create a `KafkaRebalance` resource with `remove-disks` mode.


Suggested change

Now lets try to move the data of volume 1 and volume 2 to volume 0, present on broker with ID 0. For doing that let's create a `KafkaRebalance` resource with `remove-disks` mode.

Next, let's move the data from volumes 1 and 2 to volume 0 on the broker with ID 0.

To achieve this, we create a `KafkaRebalance` resource in `remove-disks` mode.

PaulRMellor · 2025-01-07T16:17:50Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+### Additional notes
+
+1. This feature only works if JBOD storage is enabled
+2. Make sure you have more than one volume per broker else you will be prompted of not having enough volumes to move the data to.


Suggested change

2. Make sure you have more than one volume per broker else you will be prompted of not having enough volumes to move the data to.

2. Make sure you have more than one volume per broker else you will be prompted for not having enough volumes to move the data to.

PaulRMellor · 2025-01-07T16:20:52Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+kubectl get kafkarebalance my-rebalance -n myproject -o yaml
+```
+
+and you should be able to get an output like this:


Suggested change

and you should be able to get an output like this:

And you should be able to get an output like this:

PaulRMellor · 2025-01-07T16:22:43Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+
+### Additional notes
+
+1. This feature only works if JBOD storage is enabled


Suggested change

1. This feature only works if JBOD storage is enabled

1. This feature only works if JBOD storage is enabled and multiple disks are used.

PaulRMellor · 2025-01-07T16:32:28Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+
+1. This feature only works if JBOD storage is enabled
+2. Make sure you have more than one volume per broker else you will be prompted of not having enough volumes to move the data to.
+3. This endpoint does not provide `before` load since upstream Cruise Control project does not support `verbose` with this endpoint so the `loadmap` generated should only have `afterLoad` information.


Suggested change

3. This endpoint does not provide `before` load since upstream Cruise Control project does not support `verbose` with this endpoint so the `loadmap` generated should only have `afterLoad` information.

3. The optimization proposal does not show the load before optimization, it only shows the load after optimization.

This is because in upstream Cruise Control the verbose tag is not enabled with the `remove_disks` endpoint.

PaulRMellor · 2025-01-07T16:35:11Z

_posts/2024-12-19-moving-data-between-JBOD-disks-using-cruise-control.md

+## What's next
+
+We hope this blog post has provided you with a clear understanding of how you can use the `KafkaRebalance` custom resource in `remove-disks` to easily move the data between the JBOD disks. 
+If you get stuck on any step or have any doubts, you can have read about this in or documentation on [Using Cruise Control to reassign parititon on JBOD disk](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)


Suggested change

If you get stuck on any step or have any doubts, you can have read about this in or documentation on [Using Cruise Control to reassign parititon on JBOD disk](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)

If you encounter any issues or want to know more, refer to our documentation on [Using Cruise Control to reassign partitions on JBOD disks](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)

Signed-off-by: ShubhamRwt <[email protected]>

ppatierno · 2025-01-16T08:06:39Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+### Setting up the environment
+
+Let's set up a cluster to work through an example demonstrating this feature.
+During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will be making use of the following resources:


Suggested change

During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will be making use of the following resources:

During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.

ppatierno · 2025-01-16T08:06:49Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+Let's set up a cluster to work through an example demonstrating this feature.
+During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will be making use of the following resources:
+We use a Kafka resource and KafkaNodePool resources to create a KRaft cluster.


Suggested change

We use a Kafka resource and KafkaNodePool resources to create a KRaft cluster.

ppatierno · 2025-01-16T08:09:18Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+Let's see the partition replicas assigned to the volumes on the brokers using the `kafka-log-dir.sh` tool.
+```shell
+kubectl exec -n myproject -ti my-cluster-pool-a-0   /bin/bash -- bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092  --broker-list 3,4,5 --topic-list my-topic


my-cluster-pool-a-0 is not consistent with the YAML ... it should be my-cluster-controller-0 or my-cluster-broker-3 or any other right pod. There is no node pool named pool-a.

Sorry, looks like I copied the command from the other example I generated. I will fix this

It should be some helper Pod and not one of the Kafka nodes 😉

Okay, then I will add an extra step on deploying the helper pod and then this step

I think you should be able to do just something like this:

kubectl -n myproject run kafka-consumer -ti --image=quay.io/strimzi/kafka:0.45.0-kafka-3.9.0 --rm=true --restart=Never -- bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092 --broker-list 3,4,5 --topic-list my-topic

ppatierno · 2025-01-16T08:11:03Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+After the rebalance is complete, use the `kafka-log-dirs.sh` tool again to verify that the data has been moved.
+```shell
+ kubectl exec -n myproject -ti my-cluster-pool-a-0   /bin/bash -- bin/kafka-log-dirs.sh --describe --bootstrap-server my-cluster-kafka-bootstrap:9092  --broker-list 3,4,5 --topic-list my-topic


Ditto as before about my-cluster-pool-a-0

ppatierno · 2025-01-16T08:24:52Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+1. This feature only works if JBOD storage is enabled and multiple disks are used else you will be prompted for not having enough volumes to move the data to.
+2. The optimization proposal does not show the load before optimization, it only shows the load after optimization.
+3. New partition replicas might be scheduled to the disks between cleaning them up with cruise control and removing them which might lead to data loss again.


Suggested change

3. New partition replicas might be scheduled to the disks between cleaning them up with cruise control and removing them which might lead to data loss again.

3. New partition replicas might be scheduled to the disks between cleaning them up with Cruise Control and removing them which might lead to data loss again.

ppatierno · 2025-01-16T08:25:07Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+1. This feature only works if JBOD storage is enabled and multiple disks are used else you will be prompted for not having enough volumes to move the data to.
+2. The optimization proposal does not show the load before optimization, it only shows the load after optimization.
+3. New partition replicas might be scheduled to the disks between cleaning them up with cruise control and removing them which might lead to data loss again.
+4. After all replicas are moved from the specified disk, the disk may still be used by CC during rebalances and Kafka can still use it when creating topics so make sure to delete the disk manually if not required.


Suggested change

4. After all replicas are moved from the specified disk, the disk may still be used by CC during rebalances and Kafka can still use it when creating topics so make sure to delete the disk manually if not required.

4. After all replicas are moved from the specified disk, the disk may still be used by Cruise Control during rebalances and Kafka can still use it when creating topics so make sure to delete the disk manually if not required.

Signed-off-by: ShubhamRwt <[email protected]>

PaulRMellor

Thanks for the updates from the first review. I've left a few more suggestions, but looks good to me.

Regarding removing PVCs, we don't show that in the docs, but maybe we should?

PaulRMellor · 2025-01-17T11:44:15Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.
+Failing to do so could result in data loss.
+Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.


Suggested change

Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.

Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.

PaulRMellor · 2025-01-17T11:45:12Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+### Cruise Control to move data between JBOD disks
+
+This feature will allow you to move the data from one JBOD disk to another JBOD disk using the `KafkaRebalance` custom resource that we have in Strimzi.


Suggested change

This feature will allow you to move the data from one JBOD disk to another JBOD disk using the `KafkaRebalance` custom resource that we have in Strimzi.

This feature allows you to move the data from one JBOD disk to another JBOD disk using Strimzi's `KafkaRebalance` custom resource.

PaulRMellor · 2025-01-17T11:46:40Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+### Setting up the environment
+
+Let's set up a cluster to work through an example demonstrating this feature.
+During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.


Suggested change

During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.

IN the example we will see how to safely remove the JBOD disks by moving the data from one disk to another, and we will use `Kafka` and `KafkaNodePool` resources to create a KRaft cluster.

PaulRMellor · 2025-01-17T11:46:50Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+Let's set up a cluster to work through an example demonstrating this feature.
+During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.
+Then, we create a KafkaRebalance resource in remove-disks mode, specifying the brokers and volume IDs for partition reassignment.


Suggested change

Then, we create a KafkaRebalance resource in remove-disks mode, specifying the brokers and volume IDs for partition reassignment.

Then, we create a `KafkaRebalance` resource in remove-disks mode, specifying the brokers and volume IDs for partition reassignment.

PaulRMellor · 2025-01-17T11:48:19Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+You can install the Cluster Operator with any installation method you prefer.
+You can also refer to the [Strimzi documentation](https://strimzi.io/docs/operators/in-development/deploying#con-strimzi-installation-methods_str).


Suggested change

You can install the Cluster Operator with any installation method you prefer.

You can also refer to the [Strimzi documentation](https://strimzi.io/docs/operators/in-development/deploying#con-strimzi-installation-methods_str).

You can install the Cluster Operator with the installation method you prefer, which are described in the [Strimzi documentation](https://strimzi.io/docs/operators/in-development/deploying#con-strimzi-installation-methods_str).

PaulRMellor · 2025-01-17T11:55:06Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+# ...
+```
+
+If you now check the PVCs then you will see that they are not deleted.


Suggested change

If you now check the PVCs then you will see that they are not deleted.

Checking the PVCs, we see that they are not deleted.

PaulRMellor · 2025-01-17T11:55:24Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+data-2-my-cluster-broker-5       Bound    pvc-0c126dc5-863f-4e2a-96ae-1a4fef9d8839   100Gi      RWO            standard       63m
+```
+
+It is because they are not deleted by default, and you need to remove them yourself. You can delete the PVC's using the following command.


Suggested change

It is because they are not deleted by default, and you need to remove them yourself. You can delete the PVC's using the following command.

It is because they are not deleted by default, and you need to remove them yourself. You can delete the PVCs using the following command.

we don't have this step in the docs @ShubhamRwt -- should we add it?

If we are showing an example where we are trying to remove a JBOD disk, then yes but if we are just showing how to use the endpoint, then no

PaulRMellor · 2025-01-17T11:55:33Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+kubectl delete pvc data-1-my-cluster-broker-3  -n myproject 
+```
+
+You can remove the other PVC's in the same way. 


Suggested change

You can remove the other PVC's in the same way.

You can remove the other PVCs in the same way.

PaulRMellor · 2025-01-17T12:04:05Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+
+#### Additional notes
+
+1. This feature only works if JBOD storage is enabled and multiple disks are used else you will be prompted for not having enough volumes to move the data to.


Suggested change

1. This feature only works if JBOD storage is enabled and multiple disks are used else you will be prompted for not having enough volumes to move the data to.

1. This feature only works if JBOD storage is enabled and multiple disks are used, otherwise you will be prompted for not having enough volumes to move the data to.

PaulRMellor · 2025-01-17T12:05:23Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+### What's next
+
+We hope this blog post has provided you with a clear understanding of how you can use the `KafkaRebalance` custom resource in `remove-disks` to easily move the data between the JBOD disks.
+If you encounter any issues or want to know more, refer to our documentation on [Using Cruise Control to reassign partitions on JBOD disks](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)


Suggested change

If you encounter any issues or want to know more, refer to our documentation on [Using Cruise Control to reassign partitions on JBOD disks](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)

If you encounter any issues or want to know more, refer to our documentation on [Using Cruise Control to reassign partitions on JBOD disks](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str).

scholzj

Few more comments. Also, some general comments:

Kubernetes uses the term volumes - I wonder if it would be more clear if you use volumes as well instead of disks (but feel free to treat this as optional and stick with disks if you want to).
There is no fencing of the disks. So while you cleaned the 2 volumes with the rebalance, any newly created topics might still be created on these volumes. You seem to cover that in the additional notes 3 and 4. But I think it deserves its own section to make it more clear.

scholzj · 2025-01-19T20:17:14Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Strimzi.
+The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.


There seems to be a bit of a disconnect / big jump in this sentence. Maybe you can lead it out with somehting like this?

Suggested change

Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Strimzi.

The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.

Apache Kafka is a platform that provides durability and fault tolerance by storing messages on persistent volumes.

In most cases, each Kafka broker will use one persistent volume.

However, it is also possible to use multiple volumes for each broker.

This configuration is called JBOD storage.

scholzj · 2025-01-19T20:23:41Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Strimzi.
+The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.
+Using JBOD storage, you can increase the data storage capacity for Kafka nodes, which can further lead to performance improvements.
+If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.


You talk about Moving data between JBOD disks. So it would be good to include also adding volumes -even if briefly in one or two sentences and link to the docs.

Suggested change

If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.

It might happen that you need to add or remove volumes to increase or shrink the overall capacity and performance of the Kafka cluster.

When adding new volumes, you first need to add the volume and then move some of the data to to.

That can be done using the _intrabroker_ rebalance: TOTO Link to docs.

When removing volumes, you have to first safely move the data to other volumes first.

scholzj · 2025-01-19T20:24:50Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+Failing to do so could result in data loss.
+Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.
+
+### Cruise Control to move data between JBOD disks


Following on the comment above - I don't think we want to go into details for adding volumes and focus on removing them for the rest of the blog post. So I would adjust the title accordingly.

I have changed this tile to -> New remove-disks mode in KafkaRebalance now

scholzj · 2025-01-19T20:28:12Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+    - brokerId: 3
+      volumeIds: [1, 2]
+    - brokerId: 4
+      volumeIds: [1, 2]
+    - brokerId: 5
+      volumeIds: [1, 2]


MAybe you can use the same YAML formatting as you have below in the output from Kubernetes? It might help to avoid confusion between the meaning of

volumeIds: [1, 2]

and

volumeIds: - 1 - 2

among less experienced users.

ShubhamRwt · 2025-01-20T10:51:29Z

Few more comments. Also, some general comments:

Kubernetes uses the term volumes - I wonder if it would be more clear if you use volumes as well instead of disks (but feel free to treat this as optional and stick with disks if you want to).

There is no fencing of the disks. So while you cleaned the 2 volumes with the rebalance, any newly created topics might still be created on these volumes. You seem to cover that in the additional notes 3 and 4. But I think it deserves its own section to make it more clear.

@scholzj For the second pointer, I have added a separate section as Whats missing/incomplete and merged 3 and 4th point

scholzj · 2025-01-20T10:53:30Z

Few more comments. Also, some general comments:

Kubernetes uses the term volumes - I wonder if it would be more clear if you use volumes as well instead of disks (but feel free to treat this as optional and stick with disks if you want to).

There is no fencing of the disks. So while you cleaned the 2 volumes with the rebalance, any newly created topics might still be created on these volumes. You seem to cover that in the additional notes 3 and 4. But I think it deserves its own section to make it more clear.

@scholzj For the second pointer, I have added a separate section as Whats missing/incomplete and merged 3 and 4th point

But it is not incomplete or missing, or? That would suggest you add it for example in the next release. But AFAIK this is not supported in Kafka and there are no plans to change this, or?

ShubhamRwt · 2025-01-20T11:00:44Z

Few more comments. Also, some general comments:

Kubernetes uses the term volumes - I wonder if it would be more clear if you use volumes as well instead of disks (but feel free to treat this as optional and stick with disks if you want to).

There is no fencing of the disks. So while you cleaned the 2 volumes with the rebalance, any newly created topics might still be created on these volumes. You seem to cover that in the additional notes 3 and 4. But I think it deserves its own section to make it more clear.

@scholzj For the second pointer, I have added a separate section as Whats missing/incomplete and merged 3 and 4th point

But it is not incomplete or missing, or? That would suggest you add it for example in the next release. But AFAIK this is not supported in Kafka and there are no plans to change this, or?

In the upstream CC PR, they have mentioned this as missing so maybe its something that will be implemented in future but there is no timelines. To me also it sounds like some part which is missing and can be implemented in upstream CC one day

scholzj · 2025-01-20T11:04:55Z

I don't think it can be done in Cruise Control. It would need to be done in Kafka so that you can fence the disks there. I think it is important to give the users the right expectations. For me:

Incomplete suggests you are going to complete it soon -> I do not think that is the case?
Missing is strictly speaking correct. But it would be good to explain what is missing and where.

ShubhamRwt · 2025-01-20T11:26:36Z

Based on what I understood from the last comment, I will change the header to just -> What's missing and then update it with more context

Signed-off-by: ShubhamRwt <[email protected]>

scholzj

I left few more nits. But it looks good to me overall. Nice work, thanks.

scholzj · 2025-01-21T16:52:36Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+Failing to do so could result in data loss.
+Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.
+
+### New `remove-disks` mode in KafkaRebalance


If you check the preview, the formatting looks bad due to some bad styles I guess. Maybe you can just leave out the formatting?

Suggested change

### New `remove-disks` mode in KafkaRebalance

### New remove-disks mode in KafkaRebalance

scholzj · 2025-01-21T16:54:05Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+That can be done using the [_intrabroker_](https://strimzi.io/docs/operators/in-development/deploying#con-rebalance-str) rebalance.
+When removing volumes, you have to first safely move the data to other volumes first.
+Failing to do so could result in data loss.
+Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.


Maybe check with @PaulRMellor ... but I wonder if someting like this would sound better.

Suggested change

Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.

Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly.

Therefore - in Strimzi 0.45.0 - we introduced the ability to move data between the JBOD disks using Cruise Control.

scholzj · 2025-01-21T16:58:49Z

_posts/2025-01-15-moving-data-between-JBOD-disks-using-cruise-control.md

+  sessionId: 028a7dc8-8f6d-485e-8580-93225528b587
+```
+
+Now you can use the `approve` annotation to apply the generated proposal.


Should we link to the docs for more details about the approval? People wh try it for the first time might find that useful.

Blog post for moving data between JBOD disks using Cruise Control

50499b0

Signed-off-by: ShubhamRwt <[email protected]>

ShubhamRwt requested review from scholzj, ppatierno, kyguy and PaulRMellor December 19, 2024 14:26

scholzj reviewed Dec 19, 2024

View reviewed changes

ShubhamRwt commented Dec 19, 2024

View reviewed changes

scholzj reviewed Dec 19, 2024

View reviewed changes

ppatierno reviewed Dec 20, 2024

View reviewed changes

PaulRMellor reviewed Jan 7, 2025

View reviewed changes

Added suggestions by Jakub, Paolo and Paul

f19970f

Signed-off-by: ShubhamRwt <[email protected]>

ShubhamRwt force-pushed the postJBOD branch from 473d344 to f19970f Compare January 15, 2025 12:58

ppatierno reviewed Jan 16, 2025

View reviewed changes

Added suggestion by Paolo and Jakub

da71d3d

Signed-off-by: ShubhamRwt <[email protected]>

PaulRMellor approved these changes Jan 17, 2025

View reviewed changes

scholzj reviewed Jan 19, 2025

View reviewed changes

Added suggestion by Jakub and Paul

1fb9da2

Signed-off-by: ShubhamRwt <[email protected]>

scholzj approved these changes Jan 21, 2025

View reviewed changes


		### Additional notes

		1. This feature only works if JBOD storage is enabled

	If the data is not removed from the disk, and it is removed then potential data loss can happen.
	If the data is not removed from the disk, and the disk is removed then potential data loss can happen.

	Currently, moving data between the JBOD disks is done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control
	Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool which is not very user-friendly, therefore in Strimzi 0.45.0 we are introducing the ability to move data between the JBOD disks using Cruise Control


		## Cruise Control to move data between JBOD disks

		This feature will allow you to move the data between the JBOD disks using the `KafkaRebalance` custom resource that we have in Strimzi.

	This feature makes use of the `remove-disks` endpoint of Cruise Control that triggers a rebalancing operation which moves replicas, starting with the largest and proceeding to the smallest, to the remaining disks.
	This feature makes use of the `remove-disks` endpoint of Cruise Control that triggers a rebalancing operation which moves all replicas, starting with the largest and proceeding to the smallest, to the remaining disks.

	title: "Moving data between the JBOD disks using Cruise Control"
	title: "Moving data between JBOD disks using Cruise Control"

	Once you create the topic, now you can check whether the volumes have some partition replicas assigned to them or not using the `kafka-log-dir.sh` tool. Let's see the partition replicas assigned to the volumes on broker with id 0.
	Now you can check whether the volumes have some partition replicas assigned to the topics using the `kafka-log-dir.sh` tool. Let's see the partition replicas assigned to the volumes on broker with id 0.

	Now lets try to move the data of volume 1 and volume 2 to volume 0, present on broker with ID 0. For doing that let's create a `KafkaRebalance` resource with `remove-disks` mode.
	Next, let's move the data from volumes 1 and 2 to volume 0 on the broker with ID 0.
	To achieve this, we create a `KafkaRebalance` resource in `remove-disks` mode.

	2. Make sure you have more than one volume per broker else you will be prompted of not having enough volumes to move the data to.
	2. Make sure you have more than one volume per broker else you will be prompted for not having enough volumes to move the data to.

	and you should be able to get an output like this:
	And you should be able to get an output like this:

	3. This endpoint does not provide `before` load since upstream Cruise Control project does not support `verbose` with this endpoint so the `loadmap` generated should only have `afterLoad` information.
	3. The optimization proposal does not show the load before optimization, it only shows the load after optimization.
	This is because in upstream Cruise Control the verbose tag is not enabled with the `remove_disks` endpoint.

	If you get stuck on any step or have any doubts, you can have read about this in or documentation on [Using Cruise Control to reassign parititon on JBOD disk](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)
	If you encounter any issues or want to know more, refer to our documentation on [Using Cruise Control to reassign partitions on JBOD disks](https://strimzi.io/docs/operators/latest/deploying#proc-cruise-control-moving-data-str)

	During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will be making use of the following resources:
	During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.

	3. New partition replicas might be scheduled to the disks between cleaning them up with cruise control and removing them which might lead to data loss again.
	3. New partition replicas might be scheduled to the disks between cleaning them up with Cruise Control and removing them which might lead to data loss again.

	4. After all replicas are moved from the specified disk, the disk may still be used by CC during rebalances and Kafka can still use it when creating topics so make sure to delete the disk manually if not required.
	4. After all replicas are moved from the specified disk, the disk may still be used by Cruise Control during rebalances and Kafka can still use it when creating topics so make sure to delete the disk manually if not required.


		### Cruise Control to move data between JBOD disks

		This feature will allow you to move the data from one JBOD disk to another JBOD disk using the `KafkaRebalance` custom resource that we have in Strimzi.

	This feature will allow you to move the data from one JBOD disk to another JBOD disk using the `KafkaRebalance` custom resource that we have in Strimzi.
	This feature allows you to move the data from one JBOD disk to another JBOD disk using Strimzi's `KafkaRebalance` custom resource.

	During the example we will see how we can safely remove the JBOD disks by moving the data from one disk to another, and we will use Kafka and KafkaNodePool resources to create a KRaft cluster.
	IN the example we will see how to safely remove the JBOD disks by moving the data from one disk to another, and we will use `Kafka` and `KafkaNodePool` resources to create a KRaft cluster.

	Then, we create a KafkaRebalance resource in remove-disks mode, specifying the brokers and volume IDs for partition reassignment.
	Then, we create a `KafkaRebalance` resource in remove-disks mode, specifying the brokers and volume IDs for partition reassignment.

		You can install the Cluster Operator with any installation method you prefer.
		You can also refer to the [Strimzi documentation](https://strimzi.io/docs/operators/in-development/deploying#con-strimzi-installation-methods_str).

Blog post for moving data between JBOD disks using Cruise Control #469

Are you sure you want to change the base?

Blog post for moving data between JBOD disks using Cruise Control #469

Conversation

ShubhamRwt commented Dec 19, 2024 • edited Loading

Type of change

ShubhamRwt commented Dec 19, 2024

Choose a reason for hiding this comment

scholzj commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShubhamRwt Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaulRMellor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scholzj Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PaulRMellor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scholzj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShubhamRwt Jan 20, 2025 • edited Loading

ShubhamRwt commented Dec 19, 2024 •

edited

Loading

ShubhamRwt Jan 15, 2025 •

edited

Loading

scholzj Jan 16, 2025 •

edited

Loading

ShubhamRwt Jan 20, 2025 •

edited

Loading

	If you now check the PVCs then you will see that they are not deleted.
	Checking the PVCs, we see that they are not deleted.

	It is because they are not deleted by default, and you need to remove them yourself. You can delete the PVC's using the following command.
	It is because they are not deleted by default, and you need to remove them yourself. You can delete the PVCs using the following command.

	You can remove the other PVC's in the same way.
	You can remove the other PVCs in the same way.


		#### Additional notes

		1. This feature only works if JBOD storage is enabled and multiple disks are used else you will be prompted for not having enough volumes to move the data to.

		Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Strimzi.
		The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.

-Apache Kafka is a platform which provides durability and fault tolerance by storing messages on disks and JBOD storage is one of the storage configuration types supported by Strimzi.
-The JBOD data storage configuration allows Kafka brokers to make use of multiple disks.
+Apache Kafka is a platform that provides durability and fault tolerance by storing messages on persistent volumes.
+In most cases, each Kafka broker will use one persistent volume.
+However, it is also possible to use multiple volumes for each broker.
+This configuration is called JBOD storage.

-If you plan to remove a disk that contains partition replicas, the data must be safely moved to other disks first.
+It might happen that you need to add or remove volumes to increase or shrink the overall capacity and performance of the Kafka cluster.
+When adding new volumes, you first need to add the volume and then move some of the data to to.
+That can be done using the _intrabroker_ rebalance: TOTO Link to docs.
+When removing volumes, you have to first safely move the data to other volumes first.

	### New `remove-disks` mode in KafkaRebalance
	### New remove-disks mode in KafkaRebalance

	Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly, therefore in Strimzi 0.45.0 we introduced the ability to move data between the JBOD disks using Cruise Control.
	Moving data between the JBOD disks can be done using the `kafka-reassign-partitions.sh` tool, which is not very user-friendly.
	Therefore - in Strimzi 0.45.0 - we introduced the ability to move data between the JBOD disks using Cruise Control.