Error launching hive-metastore on k8 cluster on aws

[arnabaws15]

Hi there!

My k8s are installed using kops. Here is the K8s and kubectl info:
Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.24.3

I first implemented hdfs and then went to proceed to install hive-metastore. I have not overwritten any values in yaml.
Below is the current status of the pods:
NAME READY STATUS RESTARTS AGE
hdfs-datanode-0 1/1 Running 0 8m40s
hdfs-datanode-1 1/1 Running 0 8m9s
hdfs-datanode-2 1/1 Running 0 7m49s
hdfs-httpfs-67d6d56bb6-w2z69 1/1 Running 0 8m40s
hdfs-namenode-0 2/2 Running 0 8m40s
hive-metastore-0 0/1 CrashLoopBackOff 5 (72s ago) 5m34s
hive-metastore-postgresql-0 1/1 Running 0 5m34s

The error I received is the following:
2022-08-22T16:16:27,275 ERROR [main] org.apache.hadoop.hive.metastore.HiveMetaStore - Metastore Thrift Server threw an exception... org.apache.hadoop.hive.metastore.api.MetaException: java.net.UnknownHostException: hive-metastore-hdfs at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:83) ~[hive-exec-2.3.2.jar:2.3.2] at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92) ~[hive-exec-2.3.2.jar:2.3.2] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6887) ~[hive-exec-2.3.2.jar:2.3.2] at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6882) ~[hive-exec-2.3.2.jar:2.3.2] at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:7140) ~[hive-exec-2.3.2.jar:2.3.2] at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:7067) [hive-exec-2.3.2.jar:2.3.2]

Can you please advise what am I missing?

Thank you!

[cgiraldo]

Hi,
Take a look to hive-metastore values.yaml.

You need to set the conf.hiveSite.hive.metastore.warehouse.dir pointing to your hdfs.

If you set it blank (as the default) it tr to connect to "hdfs://{{.Release.Name}}-hdfs:8020/user/hive/warehouse".

[arnabaws15]

Hi Carlos,
Thank you for responding. I really appreciate it.
Looks like dependencies in this chart only specify a database and not hdfs. I deployed your hdfs chart also on the cluster. Can you please share how I would reference it in the values.yaml for hive metastore?

Thank you for your help.

[cgiraldo]

If you install hdfs with:

helm install hdfs bigdata-gradiant/hdfs

You will have a k8s Service named hdfs.

Then you have to install hive-metastore with:

helm install hive-metastore bigdata-gradiant/hive-metastore --set conf.hiveSite."hive\.metastore\.warehouse\.dir"=hdfs://hdfs:8020/user/hive/warehouse

[arnabaws15]

That worked. Thank you so much!