[core][runtime_env] Retry on read error in rt env agent client response read

[rynewang]

#45353 reports issues on runtime env agent start up process + task pressure. Some times we get on_read errors. This PR fixes by retrying on those requests. Specifically, all HTTP errors before we get a well-formed HTTP response are retried transparently with a ray log.

Also I took this chance to do some quality-of-life improvements to the runtime env agent:

• Moved forward the log initialization, so we have "process started" logs and "ready to serve traffic" logs separately.
• Allow the http client to set method.
• Explicitly set HTTP/1.1.
• Uses Beast's way to set content-length.
• Sets never expire on each step.
• Returns NotFound on resolving and connecting, returns Disconnected on on_write and on_read, returns IOError on user error (HTTP non OK).
• Retry on NotFound and Disconnected.

Added a unit test for this situation.

Fixes #45353.

[jjyao]

A high level question: is it guaranteed to be safe to retry upon on_read errors?

[hongchaodeng]

LGTM after nit

[hongchaodeng]

I have reproduced the issue and verified that it is safe to retry when on_read failed.

Here's the log from runtime env agent:

2024-06-03 20:33:53,759 INFO runtime_env_agent.py:380 -- Creating runtime env: {"_ray_commit": "{{RAY_COMMIT_SHA}}", "pip": {"packages": ["emoji"], "pip_check": false}} with timeout 600 seconds.
2024-06-03 20:33:57,751 INFO runtime_env_agent.py:428 -- Successfully created runtime env: {"_ray_commit": "{{RAY_COMMIT_SHA}}", "pip": {"packages": ["emoji"], "pip_check": false}}, the context: {"command_prefix": ["source", "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/activate", "1>&2", "&&"], "env_vars": {}, "py_executable": "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/python", "resources_dir": null, "override_worker_entrypoint": null, "java_jars": []}
2024-06-03 20:33:57,751 INFO runtime_env_agent.py:464 -- Runtime env already created successfully. Env: {"_ray_commit": "{{RAY_COMMIT_SHA}}", "pip": {"packages": ["emoji"], "pip_check": false}}, context: {"command_prefix": ["source", "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/activate", "1>&2", "&&"], "env_vars": {}, "py_executable": "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/python", "resources_dir": null, "override_worker_entrypoint": null, "java_jars": []}
2024-06-03 20:33:58,753 INFO runtime_env_agent.py:464 -- Runtime env already created successfully. Env: {"_ray_commit": "{{RAY_COMMIT_SHA}}", "pip": {"packages": ["emoji"], "pip_check": false}}, context: {"command_prefix": ["source", "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/activate", "1>&2", "&&"], "env_vars": {}, "py_executable": "/tmp/ray/session_2024-06-03_20-25-13_549115_257179/runtime_resources/pip/69cd483986f3d2762be1e312f9cd49dac02e9f97/virtualenv/bin/python", "resources_dir": null, "override_worker_entrypoint": null, "java_jars": []}

It will reply STATUS_OK with correct context.

[jjyao]

This is not accurate? since we won't wait but just retry

[rynewang]

we wait every 1s before retry

[rynewang]

Actually waits for agent_manager_retry_interval_ms_ = 100ms, until total = agent_register_timeout_ms = 10000ms = 10s and hard error out.

[rynewang]

Test timeout fixed - it turns out one previous test set some timeout-ing os env var and did not set back. Changed all such env vars to monkeypatch.