Unexpected CancelledError with aiohttp 3.9.x

[ioktl]

I've bumped aiohttp version from aiohttp[speedups]==3.8.3 to aiohttp[speedups]==3.9.0 and immediately regretted everything because of the following stack trace in the logs:

  File "/home/prod/venv/lib/python3.11/site-packages/csdk/utilities/json_rest_client.py", line 50, in request
    return await response.json()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/prod/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1161, in json
    await self.read()
  File "/home/prod/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1114, in read
    await self._wait_released()  # Underlying connection released
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/prod/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1080, in _wait_released
    await self._writer
  File "/home/prod/venv/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 609, in write_bytes
    async def write_bytes(
asyncio.exceptions.CancelledError

The version bump was the only change in our code. I looked at #7764 and #7815 as possible culprits, but didn't see anything immediately sus. This can be an issue on our side somehow brought to light by the recent changes in aiohttp but it's difficult for me to verify, since we only get CancelledErrors in production, and I can't figure out a test case in a controlled environment.

To recap:

• unexpected CancelledError on ClientResponse.json() that happens only in 3.9.0 but not in 3.8.3;
• it happens consistently (i.e. it will happen at some point);
• but it only happens in our production environment where network issues and load sizes can be unpredictable.

I'm not really sure that something's wrong with aiohttp. But if someone can give me any pointers on where to look, I'd be really grateful.

[bdraco]

It looks like cancellation is leaking upwards since it's now awaited

Please open an issue with a full trace. It will be likely hard to solve without a reproducer present as it's not something that has come up before. If you can come up with a way to make it happen in a test, opening a test that is marked xfail is a sure way to demonstrate the problem and get to a solution quickly.

[ioktl]

Yeah, I'd gladly do that if I can provide a reproducible example. This is precisely why I opened this topic in discussions -- I'm trying to build a test case to make sure this is an aiohttp issue. However, no success yet. I hoped, that someone here might have an idea on where to look or what circumstances can cause this.

[Dreamsorcerer]

Maybe it's possible to get a situation where the client is trying to send eof or whatever, but the server upon finishing sending the response just ignores it and lets the connection hang? Then a timeout or whatever happens.

If you want to try and get more info, it seems the most likely place the writer is getting cancelled is _cleanup_writer(). If you could manage to verify that it is getting cancelled there, because the writer is not done at that time, and print a traceback to figure out what caused the _cleanup_writer() to get called when the writer was unfinished, that might help figure things out.

[Dreamsorcerer]

If it is a situation like I described, then maybe we should consider just suppressing the exception there, and ensuring that the connection is forcefully closed in that situation. That might give us the best tradeoff between reliability and keep-alive performance.

[bdraco]

#8041

[Dreamsorcerer]

If you're unable to get more information as described above then maybe see if you can try the linked PR and see if it resolves the issue.

[Dreamsorcerer]

@ioktl If you're able to get some information or confirm the linked PR works before the weekend, then we may be able to include this in 3.9.2.

[ioktl]

@Dreamsorcerer I'll try by best to test the PR this week. Sorry for the delay -- I need to jump through quite a few hoops to run tests in our production environment.