Unable to serialize Retry error with msgpack

[sandeep2rawat]

On using msgpack, raising arq.Retry error after max retries raising the following error.

2024-10-17 12:34:36 07:04:36: error serializing result of 8263f14d854e4cfd9c4ee42970800238:send_webhook_task
2024-10-17 12:34:36 Traceback (most recent call last):
2024-10-17 12:34:36   File "/usr/local/lib/python3.11/site-packages/arq/jobs.py", line 261, in serialize_result
2024-10-17 12:34:36     return serializer(data)
2024-10-17 12:34:36            ^^^^^^^^^^^^^^^^
2024-10-17 12:34:36   File "/usr/local/lib/python3.11/site-packages/msgpack/__init__.py", line 36, in packb
2024-10-17 12:34:36     return Packer(**kwargs).pack(o)
2024-10-17 12:34:36            ^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-17 12:34:36   File "msgpack/_packer.pyx", line 294, in msgpack._cmsgpack.Packer.pack
2024-10-17 12:34:36   File "msgpack/_packer.pyx", line 300, in msgpack._cmsgpack.Packer.pack
2024-10-17 12:34:36   File "msgpack/_packer.pyx", line 297, in msgpack._cmsgpack.Packer.pack
2024-10-17 12:34:36   File "msgpack/_packer.pyx", line 231, in msgpack._cmsgpack.Packer._pack
2024-10-17 12:34:36   File "msgpack/_packer.pyx", line 291, in msgpack._cmsgpack.Packer._pack
2024-10-17 12:34:36 TypeError: can not serialize 'JobExecutionFailed' object

[einarwar]

Related to #309. I am experiencing the same issue myself, and haven't found a way of serializing exceptions. I am using orjson, but like msgpack it cannot serialize exceptions either.

One solution could be to convert the exception to a string and then return that one, e.g

import sys
import traceback

try:
    ... # Your code here
except MyException:
    exception_string = "".join(traceback.format_exception(*sys.exc_info()))
    return exception_string

But this is not a really good solution as you would need to manually process this string in the caller, and it is a really non-standard way of handling exceptions

[einarwar]

I found out my ArqRedis had set decode_responses=True, which was the reason i needed orjson as a custom serializer in the first place. By setting decode_responses=False (the default), the default serializer (pickle) can serialize both exceptions as well as "normal" return values