Add support for setting options at reset with VecEnv

[ReHoss]

Description

Hi, I suggest the following code to solve #1523

Basically, it is inspired by #1486 logic for seeds.

Note, only one option dictionary is duplicated to be passed to all VecEnv environments when done or truncated is reached.

Change Details

I followed the logic from #1486 where the options dictionary are stored in a list whose size is the number of environments in the VecEnv.

However, I had to update the make_vec_env function signature with an options argument.

I updated all the tests signatures for reset and added one wrong_reset method.

About pytest

However, running make pytest the following errors triggered:

test_fps_logger[A2C] fails

In fact my Ubuntu 22 laptop with i9 proc was in mode eco, so the computational power was limited.
With standard power the fps test passed. Should this test be made easier?

However, I still have a rendering problem ? Note my nvidia card is deactivated.

tests/test_vec_envs.py::test_render[DummyVecEnv] make: *** [Makefile:5: pytest] Error 1

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

closes #1523

Types of changes

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have opened an associated PR on the SB3-Contrib repository (if necessary)
• I have opened an associated PR on the RL-Zoo3 repository (if necessary)
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[ReHoss]

I did not change the env_checker

stable-baselines3/stable_baselines3/common/env_checker.py

Lines 375 to 404 in d68ff2e

	def check_env(env: gym.Env, warn: bool = True, skip_render_check: bool = True) -> None:
	"""
	Check that an environment follows Gym API.
	This is particularly useful when using a custom environment.
	Please take a look at https://gymnasium.farama.org/api/env/
	for more information about the API.
	It also optionally check that the environment is compatible with Stable-Baselines.
	:param env: The Gym environment that will be checked
	:param warn: Whether to output additional warnings
	mainly related to the interaction with Stable Baselines
	:param skip_render_check: Whether to skip the checks for the render method.
	True by default (useful for the CI)
	"""
	assert isinstance(
	env, gym.Env
	), "Your environment must inherit from the gymnasium.Env class cf. https://gymnasium.farama.org/api/env/"
	# ============= Check the spaces (observation and action) ================
	_check_spaces(env)
	# Define aliases for convenience
	observation_space = env.observation_space
	action_space = env.action_space
	try:
	env.reset(seed=0)
	except TypeError as e:
	raise TypeError("The reset() method must accept a `seed` parameter") from e

Do we want to check if the signature also contains options ?

[araffin]

Hello,
thanks for the PR =), I'm taking some days off, will come back to you mid next week normally ;)

[araffin]

However, I had to update the make_vec_env function signature with an options argument.

I don't think that should be needed.

Do we want to check if the signature also contains options ?

No, compared to seed, options is not critical but a nice-to-have feature.
So, it should be optional (so should be passed to the reset only if present).
Also, I think it would make sense to allow to pass a list of dict instead of a single dict.

[ReHoss]

However, I had to update the make_vec_env function signature with an options argument.

I don't think that should be needed.

As #1486 augment make_vec_env with a seed initialisation (see [*]), I applied the same logic with the options (despite the unique options is diffused to all environments for now).
Why would we not do that here? To avoid adding an optional parameter?

[*] : https://github.com/DLR-RM/stable-baselines3/pull/1486/files#diff-d5121994cbf23725faf52c1bbc0e5b1baeff4ab35ce176eefaf24b63c09f9cb0R125-R128

Do we want to check if the signature also contains options ?

No, compared to seed, options is not critical but a nice-to-have feature. So, it should be optional (so should be passed to the reset only if present). Also, I think it would make sense to allow to pass a list of dict instead of a single dict.

Alright, In the context of stable-baselines3 the reset methods must have seed as argument while options is optional.

I'll be back by the end off August:)

[kmcminn]

quick +1 for this feature. Also can we make it easier to extend step() also with options/kwargs?