Pekko Connectors for pulling data from HDFS to putting into S3

[Susmit07]

Hello Developer community,

We have a requirement oc pulling data from HDFS to putting into S3.

We are exploring Pekko connectors, do you think it's an ideal use case to use Pekko connectors?

Hdfs connectors to pull data from hdfs and s3 connectors to push to s3 bucket

Requesting your help

Thanks.

[pjfanning]

@Susmit07 I'll leave this open just in case some developers see this and can share some insights.

In practice, Apache Pekko is a volunteer org with a small active community. We don't really have much bandwidth for answering general questions like this. There are 2 connectors for S3 and HDFS as you describe and they are there for you to try out. If you hit specific issues, please get in touch.

[Susmit07]

@pjfanning we have a requirement of pulling parquet file whenever its available in the specified hdfs path, and we need to be sure on the exactly once delivery ( no data duplication on the sink side), is it feasible using pekko connectors ?

[pjfanning]

These are featured that Pekko streams and connectors support. pekko.apache.org has our docs.

[raboof]

Generally, the trick to exactly once delivery across different systems is to make sure the sink side is idempotent: in other words, it's essentially impossible to avoid 'retries' of failed write operations, but that is no problem because instead you make sure those retries don't lead to duplication.

Whether that's possible in your case depends on if you have any particular requirements on what things should look like on the S3 side: if you can set things up so that retries 'overwrite' the data from the previous attempt, or can check whether you have already successfully uploaded something, you should be in good shape.

[Susmit07]

but that is no problem because instead you make sure those retries don't lead to duplication.

that's not feasible in our case, the only approach can be thought is using s3 multipart upload all-or-none ..

[raboof]

if you cannot make sure retries don't lead to duplication (either by querying s3 to see whether the data is already there, or by using s3 operations that would 'overwrite' existing upload without causing duplication), then you cannot have exactly once delivery. Not with Pekko, but not with any other tools either: because in that case, for an aborted upload, you cannot reliably determine whether the upload has happened or needs to be repeated.

[Susmit07]

@pjfanning @raboof One more question suppose we deploy the code in a cluster mode, and both the process will be watching the same directories, there can be race condition where both the process may read the same file, so we will need a distribute lock mechanism like ZK / redis etc.

Does pekko as a framework has any ways to lock across cluster nodes.

I know there is LWWMap (Last-Write-Wins Map) which is a part of Pekko Distributed Data, which provides CRDTs (Conflict-Free Replicated Data Types) for managing distributed, replicated state in a cluster, but while doing a POC i found nondeterministic results

[raboof]

What is the reason for introducing clustering here? at first glance it seems to add a lot of complexity for unclear benefits.

My first strategy would be to have a static number of nodes, and configure each node to process a predefined subset of resources in HDFS (e.g. by prefix/postfix).

If you need dynamic scaling, you could consider creating a Pekko Cluster using Cluster Sharding.

I'm not sure what you mean by "nondeterministic results", but LWWMap should definitely be eventually consistent.

[Susmit07]

If you need dynamic scaling, you could consider creating a Pekko Cluster using Cluster Sharding.

The connector should be able to pull data from hdfs and ceph (s3 like object store) the file format will be parquet. For better scalability and parallelism we are thinking to go deploy the application code within 2 K8s pods (to start with).

https://nightlies.apache.org/pekko/docs/pekko-management/main-snapshot/docs/kubernetes-deployment/forming-a-cluster.html

is it the correct document to point to or the approach for dynamic scaling is wrong

Lastly thanks for the replies and the discussion