sys.setrecursionlimit no longer affecting pickle.dump

[Jiehong]

Bug report

Bug description:

This is a follow up of #112282 (updating doc).
It's also a different case of #128544 (import chains impacted).

In particular, on top of the doc no longer being correct for sys.setrecursionlimit, the regression needs to be fixed.

Note: this works in python 3.11 or below.

import pickle
import sys

nested_object = current = {}
for i in range(10000):
    current['nested'] = {}
    current = current['nested']

sys.setrecursionlimit(3750000)
x = pickle.dumps(nested_object)

Result of the run:

▶▶ python3.11 test-issue.py
# runs ok

▶▶ python3.12 test-issue.py
Traceback (most recent call last):
  File "~/test-issue.py", line 10, in <module>
    x = pickle.dumps(nested_object)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while pickling an object

Impact: all programs trying to pickle data that is too nested cannot be ported to python 3.12 or 3.13 so far

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux, macOS, Windows

[markshannon]

With this program:

import pickle
import sys

nested_object = current = {}
for i in range(100000):
    next = {}
    current['nested'] = next
    current = next

for r in range(5000, 1000000, 5000):
    try:
        print("Trying with recursion limit", r)
        sys.setrecursionlimit(r)
        x = pickle.dumps(nested_object)
    except RecursionError:
        print("Recursion error")

With Python 3.11, I get:

Trying with recursion limit 5000
Recursion error
...
Trying with recursion limit 115000
Recursion error
Trying with recursion limit 120000
Segmentation fault (core dumped)

With 3.12, I get:

Trying with recursion limit 5000
Recursion error
...
Trying with recursion limit 995000
Recursion error

It is impossible to pickle an arbitrary long chain of dicts on any version of Python, there simply isn't enough C stack space.
Python 3.11 crashes. Python 3.12 does not. This doesn't seem like a regression.

[Jiehong]

Python 3.11 crashes. Python 3.12 does not. This doesn't seem like a regression.

For your particular program, indeed. I'm not saying pickle dumping is perfect for all structures. And for the program I provided, the result is opposite of yours, and this is a regression.

We've got a program in production that does pickle.dump, and without changing the program, it works fine from python 3.8 to python 3.11, yet it does not with python 3.12 and later with this exact message.

This, by definition, is a regression.

[picnixz]

I'm categorizing this one as a pickle bug for now, although I'm not sure if it's directly related to pickle or if it's related to something else (maybe the recursion limit is reached before due to some other changes in the interpreter).

cc @serhiy-storchaka

[serhiy-storchaka]

It is impossible to pickle an arbitrary long chain of dicts on any version of Python

But it was possible to pickle a chain of length 50000, and now it is not possible to pickle even a chain of length 5000. This is a regression.

It is interesting that a chain of length 4990 can be pickled with arbitrary small recursion limit, but pickling a chain of length 5000 fails for any recursion limit. This is suspicious.

I do not think that this issue is a pickle-specific.