Error: "No default __reduce__ due to non-trivial __cinit__"

[madiator]

If you have a separate parse_func where you add type annotation to the argument, you get this strange error.

e.g.

from bespokelabs import curator
from datasets import Dataset
from pydantic import BaseModel, Field
from typing import List

# Create a dataset object for the topics you want to create the poems.
topics = Dataset.from_dict({"topic": [
    "Urban loneliness in a bustling city",
    "Beauty of Bespoke Labs's Curator library"
]})

# Define a class to encapsulate a list of poems.
class Poem(BaseModel):
    poem: str = Field(description="A poem.")

class Poems(BaseModel):
    poems_list: List[Poem] = Field(description="A list of poems.")

# removing the type annotation for `poems` works fine.
def parse_func(row, poems: Poems):
    return [
        {"topic": row["topic"], "poem": p.poem} for p in poems.poems_list
    ]

# We define a Prompter that generates poems which gets applied to the topics dataset.
poet = curator.Prompter(
    prompt_func=lambda row: f"Write two poems about {row['topic']}.",
    model_name="gpt-4o-mini",
    response_format=Poems,
    parse_func=parse_func,
)
poem = poet(topics)
print(poem.to_pandas())

Error:

TypeError                                 Traceback (most recent call last)
[<ipython-input-4-38125e72f173>](https://localhost:8080/#) in <cell line: 35>()
     33     parse_func=parse_func,
     34 )
---> 35 poem = poet(topics)
     36 print(poem.to_pandas())

136 frames
/usr/local/lib/python3.10/dist-packages/zmq/backend/cython/socket.cpython-310-x86_64-linux-gnu.so in zmq.backend.cython.socket.Socket.__reduce_cython__()

TypeError: no default __reduce__ due to non-trivial __cinit__

[vutrung96]

I cannot reproduce this error. I only get warnings

/home/trung/.cache/pypoetry/virtualenvs/bespokelabs-curator-H3Ru9elj-py3.11/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot locate reference to <class '__main__.Poem'>.
  StockPickler.save(self, obj, save_persistent_id)
/home/trung/.cache/pypoetry/virtualenvs/bespokelabs-curator-H3Ru9elj-py3.11/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot pickle <class '__main__.Poem'>: __main__.Poem has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
/home/trung/.cache/pypoetry/virtualenvs/bespokelabs-curator-H3Ru9elj-py3.11/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot locate reference to <class '__main__.Poems'>.
  StockPickler.save(self, obj, save_persistent_id)
/home/trung/.cache/pypoetry/virtualenvs/bespokelabs-curator-H3Ru9elj-py3.11/lib/python3.11/site-packages/dill/_dill.py:414: PicklingWarning: Cannot pickle <class '__main__.Poems'>: __main__.Poems has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)

[vutrung96]

Seems like the warning comes from: uqfoundation/dill#650

[vutrung96]

Per uqfoundation/dill#650 (comment), it seems like this only occurs when the class is declared in main, like @shreyaspimpalgaonkar said.

[vutrung96]

Given that this only throws a warning and is restricted to a certain scenario, I'll lower the priority.