Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Grafana] Add a Cluster variable to the Grafana Dashboard to enable filtering of different RayClusters #2685

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

win5923
Copy link
Contributor

@win5923 win5923 commented Dec 24, 2024

Why are these changes needed?

The current Grafana Dashboard aggregates metrics from all Ray clusters and summarizes them. Add a Cluster variable to Grafana dashboard to enable viewing metrics for individual RayClusters.

Ref: #2502 (comment)

TODO

Before:

  • Ray core dashboard:
    image
  • Ray data dashboard:
    image

After:

  • Ray core dashboard:
    image

image

  • Ray data dashboard:
    image

Related issue number

Closes #2502

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

@win5923 win5923 force-pushed the grafana/ray_io_cluster branch from ad08687 to 0e7250a Compare December 24, 2024 14:08
@win5923
Copy link
Contributor Author

win5923 commented Dec 24, 2024

@kevin85421 PTAL

@win5923 win5923 force-pushed the grafana/ray_io_cluster branch from 0e7250a to a6b8183 Compare December 24, 2024 14:27
@win5923 win5923 changed the title [Metrics] Add a RayCluster variable to the Grafana Dashboard to nable filtering of different RayClusters [Metrics] Add a RayCluster variable to the Grafana Dashboard to enable filtering of different RayClusters Dec 24, 2024
@win5923 win5923 changed the title [Metrics] Add a RayCluster variable to the Grafana Dashboard to enable filtering of different RayClusters [Grafana] Add a RayCluster variable to the Grafana Dashboard to enable filtering of different RayClusters Dec 25, 2024
@kevin85421 kevin85421 self-assigned this Dec 25, 2024
@win5923 win5923 marked this pull request as draft December 26, 2024 02:51
@win5923
Copy link
Contributor Author

win5923 commented Dec 26, 2024

Ray places the JSON file in /tmp/ray/session_latest/metrics/grafana/dashboards/ inside the head pod. I will first modify the JSON file generated by Ray, and then update here.

image

@win5923 win5923 marked this pull request as ready for review December 26, 2024 05:37
@win5923 win5923 force-pushed the grafana/ray_io_cluster branch from a6b8183 to cabb01d Compare December 29, 2024 14:14
@win5923 win5923 force-pushed the grafana/ray_io_cluster branch from cabb01d to 0536234 Compare December 30, 2024 15:48
@win5923 win5923 marked this pull request as draft December 31, 2024 00:57
@win5923 win5923 marked this pull request as ready for review January 8, 2025 15:51
@win5923
Copy link
Contributor Author

win5923 commented Jan 8, 2025

According to ray-project/ray#49498. We can use the Cluster variable to filter metrics from different Ray Clusters.

cc @kevin85421
image
image

@win5923 win5923 force-pushed the grafana/ray_io_cluster branch from dd6644f to 6088351 Compare January 8, 2025 15:54
@win5923 win5923 changed the title [Grafana] Add a RayCluster variable to the Grafana Dashboard to enable filtering of different RayClusters [Grafana] Add a Cluster variable to the Grafana Dashboard to enable filtering of different RayClusters Jan 9, 2025
@kevin85421 kevin85421 removed the pause label Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Multiple RayCluster CRs share the same Grafana
2 participants