core: Handle unterminated escape character when parsing partial JSON

[balvisio]

Description
Currently, when parsing a partial JSON, if a string ends with the escape character, the whole key/value is removed. For example:

>>> from langchain_core.utils.json import parse_partial_json
>>> my_str = '{"foo": "bar", "baz": "qux\\'
>>> 
>>> parse_partial_json(my_str)
{'foo': 'bar'}

My expectation (and with this fix) would be for parse_partial_json() to return:

>>> from langchain_core.utils.json import parse_partial_json
>>> 
>>> my_str = '{"foo": "bar", "baz": "qux\\'
>>> parse_partial_json(my_str)
{'foo': 'bar', 'baz': 'qux'}

Notes:

1. It could be argued that current behavior is still desired.
2. I have experienced this issue when the streaming output from an LLM and the chunk happens to end with \\
3. I haven't included tests. Will do if change is accepted.
4. This is specially troublesome when this function is used by
   

   langchain/libs/core/langchain_core/output_parsers/transform.py

   Line 111 in 187131c

   def _transform(self, input: Iterator[Union[str, BaseMessage]]) -> Iterator[Any]:

since what happens is that, for example, if the received sequence of chunks are: {"foo": "b , ar\\ :

Then, the result of calling self.parse_result() is:

{"foo": "b"}

and the second time:

{}

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jan 8, 2025 2:27am

[eyurtsev]

Hi @balvisio thanks for looking into this. Could you add a unit test for this scenario?

[balvisio]

@eyurtsev Done

[efriis]

makes sense