json.dumps() lacks information about RecursionError related to default function

[KrzysztofNazarewski]

BPO	30732
Nosy	@rhettinger, @etrepum, @ezio-melotti, @serhiy-storchaka
PRs	bpo-30732: json.dumps() lacks information about RecursionError relate… #2856 bpo-30732: json.dumps() lacks information about RecursionError relate… #2857

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-06-22.09:38:57.037>
labels = ['3.7', 'type-feature', 'library']
title = 'json.dumps() lacks information about RecursionError related to default function'
updated_at = <Date 2017-07-31.15:08:28.815>
user = 'https://bugs.python.org/KrzysztofNazarewski'

bugs.python.org fields:

activity = <Date 2017-07-31.15:08:28.815>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2017-06-22.09:38:57.037>
creator = 'Krzysztof Nazarewski'
dependencies = []
files = []
hgrepos = []
issue_num = 30732
keywords = []
message_count = 2.0
messages = ['296620', '299561']
nosy_count = 5.0
nosy_names = ['rhettinger', 'bob.ippolito', 'ezio.melotti', 'serhiy.storchaka', 'Krzysztof Nazarewski']
pr_nums = ['2856', '2857']
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue30732'
versions = ['Python 3.7']

Linked PRs

• gh-74917: Raise TypeError in the JSON encoder if the default function returns the same type #113458

[KrzysztofNazarewski]

RecursionErrors related to json.dumps' default argument give no information whatsoever about the underlying issue.

Example:

$ /usr/bin/python3 -c "import json;from decimal import Decimal;json.dumps(Decimal(),default=lambda v:round(v, 8))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "<string>", line 1, in <lambda>
  File "/usr/lib/python3.6/_pydecimal.py", line 1919, in __round__
    return self.quantize(exp)
  File "/usr/lib/python3.6/_pydecimal.py", line 2603, in quantize
    return ans._fix(context)
  File "/usr/lib/python3.6/_pydecimal.py", line 1720, in _fix
    if not self:
  File "/usr/lib/python3.6/_pydecimal.py", line 852, in __bool__
    return self._is_special or self._int != '0'
RecursionError: maximum recursion depth exceeded in comparison

Details:
I have encountered issue when porting my code from Python 2 to 3.
Traceback shows depth of 6, but says recursion was already reached.
My first idea was that recursion counter was messed up, but it was not the case.
Recursion limit was indeed reached because round() in Python 3 is returning Decimal and then applies default infinitely.
The part of recursing code was inside C so it was not displayed in the traceback.

Summing up it took me over 2 hours to determine what was wrong with the code.

Fix ideas:

• do not call default more than once on the same root object (might be problematic to implement)
• raise an error if new object is equal to the previous (partially resolving issue since custom objects might not implement equality operator)
• add a flag to raise an error when new object's class is equal to the previous (might give false positives hence the flag)

[serhiy-storchaka]

simplejson behaves the same.

I thought default() was allowed to return only instances of types directly supported by JSON encoder (type(None), bool, int, float, list, tuple, dict). Is there a reason of allowing to call default() more than once on the same root object?

[encukou]

This is a duplicate of the newer #122163, which has a merged fix: adding the context to __notes__.

$ python3.14 -c "import json;from decimal import Decimal;json.dumps(Decimal(),default=lambda v:round(v, 8))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import json;from decimal import Decimal;json.dumps(Decimal(),default=lambda v:round(v, 8))
                                            ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/encukou/dev/cpython/Lib/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
          ~~~~~~^^^^^
  File "/home/encukou/dev/cpython/Lib/json/encoder.py", line 200, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/encukou/dev/cpython/Lib/json/encoder.py", line 261, in iterencode
    return _iterencode(o, 0)
  File "<string>", line 1, in <lambda>
    import json;from decimal import Decimal;json.dumps(Decimal(),default=lambda v:round(v, 8))
                                                                                  ~~~~~^^^^^^
RecursionError: maximum recursion depth exceeded while calling a Python object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
when serializing decimal.Decimal object
[...etc.]