Introduce manually managed partitions #56
Conversation
ylegat
force-pushed
the
allow-manually-managed-partitions
branch
2 times, most recently
from
05f5fde to
afe42f3
Compare
The data used as a partition key is not always of the greatest quality. For instance, a timestamp column could contain some invalid dates referring to year -1. Ideally these values should be cleaned, but it is not always an option in a system where data is treated as immutable. One way to deal with this situation is to add a default partition that will be in charge of containing data that cannot be assigned to any of the partitions already defined. Another solution, when the range of invalid values is known, is to move this data inside one or several dedicated partitions. Both solutions involve the creation of partitions that cannot be managed by the partition manager, because their range break the policy defined for the other partitions. For this reason we propose to introduce a new configuration property: `manuallyManagedPartitions`. It indicates which partitions should be ignored by the partition manager when checking or cleaning up partitions tables. Signed-off-by: Yohan Legat <[email protected]>
ylegat
force-pushed
the
allow-manually-managed-partitions
branch
from
afe42f3 to
d908602
Compare
Minimum allowed line rate is |
|
Hi @ylegat, thank you for your contribution 👍 Before going into the details of the implementation, I'd like to better understand the needs behind this MR. I understand that the PPM is quite restrictive in how it allows you to setup your partitions. Our initial intention was to help apply best practices by leaving almost no flexibility in how the partitions are setup. An example of this is that we intentionally do not support a default partition because this is not considered a good practice. Whilst it protects you against holes within your partitions, it can silently start accumulating data and slowly but surely degrade your performance which defies the initial purpose of partitioning. Now, this being said, can you give me a bit more context related to having corrupted data? You mention having incorrect dates. I am not clear how this is blocking you in using the partitions generated by PPM? I understand the requirement of immutability, but how is this data usable if not cleaned? 🤔 I honestly would rather challenge our ability to clean the data instead of quickly adding the ability to PPM to handle manually created partition. There are a few reasons I can think of that makes me consider against adding this capability:
|
|
As discussed together, your partitioning use case is not meant to be supported by PPM. We identified together that hash partitioning is more suitable to your needs which is not the "common" use case that PPM aims to cover. |
The data used as a partition key is not always of the greatest quality. For instance, a timestamp column could contain some invalid dates referring to year -1. Ideally these values should be cleaned, but it is not always an option in a system where data is treated as immutable.
One way to deal with this situation is to add a default partition that will be in charge of containing data that cannot be assigned to any of the partitions already defined. Another solution, when the range of invalid values is known, is to move this data inside one or several dedicated partitions.
Both solutions involve the creation of partitions that cannot be managed by the partition manager, because their range break the policy defined for the other partitions.
For this reason we propose to introduce a new configuration property:
manuallyManagedPartitions. It indicates which partitions should be ignored by the partition manager when checking or cleaning up partitions tables.