[RayCluster][Feature] add GcsFaultToleranceOptions to the RayCluster CRD [2/N]

[rueian]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, enabling RayCluster GCS FT is cumbersome given that users need to update many settings that are scattered around in their CR yaml files. The new GcsFaultToleranceOptions field in RayCluster CRD has been merged (#2715). The field serves as a new central place containing all configs for enabling GCS FT. Now we need to implement the actual logic to enable the feature according to the new field.

Our goal is when the new field is set, the KubeRay should configure the RayCluster with GCS FT enabled. That includes setting corresponding Pods annotations, environment variables, and RayStartParams. See the doc for more details.

We plan to implement it in 2 separate PRs:

1. Set RayFTEnabledAnnotationKey and RayExternalStorageNSAnnotationKey pod annotations according to the GcsFaultToleranceOptions.
2. Set the RAY_REDIS_ADDRESS environment variable to Head Pod according to the GcsFaultToleranceOptions.
3. Set the REDIS_PASSWORD environment variable and the corresponding RayStartParam to Head Pod according to the GcsFaultToleranceOptions.
4. Set the REDIS_USERNAME environment variable and the corresponding RayStartParam to Head Pod according to the GcsFaultToleranceOptions as well as adding the support of the REDIS_USERNAME to the usage of the cleanup_redis_storage.

This issue targets the first part: Setting pod annotations.

Related issues

#2695

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[rueian]

#2721 has addressed the 1. and 2. parts.

[fscnick]

#2731 has addressed the 3. parts.