Docs: note requirement to normalise unicode identifiers passed to globals() and locals()

[outofculture]

BPO	42680
Nosy	@ericvsmith, @ezio-melotti, @stevendaprano, @outofculture

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-12-19.00:27:18.270>
labels = ['3.8', 'type-bug', '3.7', '3.9', 'expert-unicode']
title = 'unicode identifiers not accessible or assignable through globals()'
updated_at = <Date 2021-01-20.10:46:27.879>
user = 'https://github.com/outofculture'

bugs.python.org fields:

activity = <Date 2021-01-20.10:46:27.879>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2020-12-19.00:27:18.270>
creator = 'outofculture'
dependencies = []
files = []
hgrepos = []
issue_num = 42680
keywords = []
message_count = 5.0
messages = ['383336', '383338', '383340', '383348', '383462']
nosy_count = 4.0
nosy_names = ['eric.smith', 'ezio.melotti', 'steven.daprano', 'outofculture']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42680'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[outofculture]

This behavior is best described by the code below:

>>> meow = 1
>>> 'meow' in globals()
True
>>> µmeow = 1e-6
>>> 'µmeow' in globals()
False
>>> globals()['woof'] = 1
>>> woof
1
>>> globals()['µwoof'] = 1e-6
>>> µwoof
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'μwoof' is not defined
>>> import sys
>>> sys.getdefaultencoding()
'utf-8'
>>> [(k, bytes(k, 'utf-8')) for k in globals()]
..., ('μmeow', b'\xce\xbcmeow'), ('µwoof', b'\xc2\xb5woof')]
>>> 'µ'.encode('utf-8')
b'\xc2\xb5'

Testing was done on linux and windows, variously using 3.6.12, 3.7.6, 3.8.6 and 3.9.0+.

[outofculture]

Oh, I just gave a cursory using locals(), and the same misbehavior is present.

A workaround, for anyone needing to assign or access unicode globals, is to use exec, e.g. exec("µmeow = 1e-6", globals()).

[stevendaprano]

I'm pretty sure this is not a bug, but is working as designed.

The interpreter normalises unicode identifiers, but key lookup in the dict does not.

Sorry, I don't have time right now to give a more detailed answer, but there are two distinct mu characters:

μ U+03BC
µ U+00B5

and my prediction is that the identifier is normalised to the first, the actual Greek mu, but you are looking up the second, the micro sign.

(I'll be able to give a longer response in a couple of hours, if still needed.)

[outofculture]

Ah! So then the proper code for me would be e.g.:

>>> globals()[unicodedata.normalize("NFKC", "µmeow")]
1e-06

Yes, it's clear when I read https://www.python.org/dev/peps/pep-3131/ that the normalization is going to happen. Is it also worth adding a note in the documentation for globals() and locals()? That's the first place I looked to try to find out what was wrong.

[ericvsmith]

I think documenting this with globals() and locals() is a good idea. That's the place this is most likely to trip someone up.