Help delta-rs migrate?

[ion-elgreco]

I am wondering whether you guys could help us migrate stuff to kernel quicker?

Currently kernel and delta-rs is evolving side by side which is wasted effort imho

[zachschuermann]

Hi @ion-elgreco! I would love to help delta-rs migrate. Now that the kernel has matured I'll begin to prioritize our integrations as much as I can. We've hopefully got some changes landing (checkpointing, a continuously improving log replay, etc.) which will make delta-rs' life easier :) - our weekly sync is on thursdays if you'd like to join to discuss some work that we can share? or happy to set aside some time for us to sync!

[riziles]

@ion-elgreco and @zachschuermann , one comment on this:

I believe Polars and DuckDB both use delta-kernel-rs under the hood to read delta tables; however, both of them fail to read a delta table stored on a mapped network drive on Windows. They both throw the error "Unable to convert URL "file:///path/to/deltalake/part-00001-e595e40e-0357-4132-bb8e-693d76080f1b-c000.snappy.parquet" to filesystem path"

delta-rs does not have to seem to have this issue, but I haven't yet been able to ascertain why it works in one package but not the other.

[ion-elgreco]

Hi @ion-elgreco! I would love to help delta-rs migrate. Now that the kernel has matured I'll begin to prioritize our integrations as much as I can. We've hopefully got some changes landing (checkpointing, a continuously improving log replay, etc.) which will make delta-rs' life easier :) - our weekly sync is on thursdays if you'd like to join to discuss some work that we can share? or happy to set aside some time for us to sync!

Can you send me the link for this on slack? :) I'll see if I can join at those time slots

@ion-elgreco and @zachschuermann , one comment on this:

I believe Polars and DuckDB both use delta-kernel-rs under the hood to read delta tables; however, both of them fail to read a delta table stored on a mapped network drive on Windows. They both throw the error "Unable to convert URL "file:///path/to/deltalake/part-00001-e595e40e-0357-4132-bb8e-693d76080f1b-c000.snappy.parquet" to filesystem path"

delta-rs does not have to seem to have this issue, but I haven't yet been able to ascertain why it works in one package but not the other.

Pola-rs uses delta-rs not delta-kernel-rs, and dispatches the reading to their own parquet reader. Any issues with their reader should be brought up with them