Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RLlib] EnvRunners fix min/max-metrics window sizes (from inf to config.metrics_num_episodes_for_smoothing). #45575

Merged
merged 14 commits into from
May 28, 2024
Merged

[RLlib] EnvRunners fix min/max-metrics window sizes (from inf to config.metrics_num_episodes_for_smoothing). #45575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
c748df8
Changed comment.
simonsays1980 May 10, 2024
6409007
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 13, 2024
d2f9030
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 14, 2024
a3416a8
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 15, 2024
8582ad9
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 16, 2024
b565f34
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 21, 2024
c0eed1f
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 22, 2024
341cb95
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 22, 2024
b76807f
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 24, 2024
af9c9e9
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 27, 2024
646a5fc
Added the user-defined windows also to min/max values in 'EnvRunner's…
simonsays1980 May 27, 2024
e422c42
Merge branch 'master' of https://github.com/ray-project/ray
simonsays1980 May 28, 2024
3a938dc
Merge branch 'master' into env-runners-cleanup-metrics-window-sizes
simonsays1980 May 28, 2024
14102de
Added same logic for 'MultiAgentEnvRunner'.
simonsays1980 May 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions rllib/env/multi_agent_env_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -901,11 +901,13 @@ def _log_episode_metrics(self, length, ret, sec, agents=None, modules=None):
EPISODE_RETURN_MIN: ret,
},
reduce="min",
window=self.config.metrics_num_episodes_for_smoothing,
)
self.metrics.log_dict(
{
EPISODE_LEN_MAX: length,
EPISODE_RETURN_MAX: ret,
},
reduce="max",
window=self.config.metrics_num_episodes_for_smoothing,
)
8 changes: 4 additions & 4 deletions rllib/env/single_agent_env_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -818,7 +818,7 @@ def _log_episode_metrics(self, length, ret, sec):
)

# For some metrics, log min/max as well.
self.metrics.log_value(EPISODE_LEN_MIN, length, reduce="min")
self.metrics.log_value(EPISODE_RETURN_MIN, ret, reduce="min")
self.metrics.log_value(EPISODE_LEN_MAX, length, reduce="max")
self.metrics.log_value(EPISODE_RETURN_MAX, ret, reduce="max")
self.metrics.log_value(EPISODE_LEN_MIN, length, reduce="min", window=win)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! That's why I always felt that min/max stats were lifetime (rather than win=100) :D
Makes perfect sense. Thanks for this fix @simonsays1980 !

self.metrics.log_value(EPISODE_RETURN_MIN, ret, reduce="min", window=win)
self.metrics.log_value(EPISODE_LEN_MAX, length, reduce="max", window=win)
self.metrics.log_value(EPISODE_RETURN_MAX, ret, reduce="max", window=win)
Loading