HDFS data node hostname issue

[bugzyz]

Hi! We are using the hdfs service outside of the kubernetes cluster, so we are facing the HDFS data node host name issue.

Since the name node only return the internal data node hostname/ip to service outside the kubernetes cluster, so that external service cannot access to the data node through the hostname/ip. Can anyone help on this scenario? Thanks!

[cgiraldo]

We don't have yet support for using hdfs outside kubernetes cluster. To do that, we should integrate the hadoop internals to kubernetes and this is not a direct task. We need to modify the hostname published by the datanodes, create a NodePort or Host port for each individual datanode, etc.

If you need to access hdfs from outside, you can try the httpfs service with a NodePort or even and Ingress.

[imtiny]

The current chart for hdfs seems like only support to enable ingress or not.

I tried to upload a large file(4GB) to hdfs through nginx ingress to httpfs, the client always raise exceptions...

I've spent a lot of time to configure the nginx ingress to support large file uploading, to support transfer-encoding: chunked, but failed.

Finally I change the httpfs-svc.yaml to NodePort, and successfully uploaded the large file.