Problems on Windows with username or hostname containing non-ASCII characters

[pekkaklarck]

When organizing a Python training recently, one participant failed to use pip after a fresh Python 2.7.11 installation on Windows. Quick investigation showed that the reason problem was ä in her username. We failed to workaround that even by creating a new account and needed to use python setup.py install instead.

I now tried to reproduce the problem on my virtual machine. My main account there has only ASCII characters in the username but I created another for testing purposes. Clearly everything is not correct:

C:\Users\Ürjö>pip
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pip.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\pip\__init__.py", line 210, in main
    cmd_name, cmd_args = parseopts(args)
  File "c:\python27\lib\site-packages\pip\__init__.py", line 165, in parseopts
    parser.print_help()
  File "c:\python27\lib\optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
  File "c:\python27\lib\optparse.py", line 1650, in format_help
    result.append(self.format_option_help(formatter))
  File "c:\python27\lib\optparse.py", line 1633, in format_option_help
    result.append(group.format_help(formatter))
  File "c:\python27\lib\optparse.py", line 1114, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "c:\python27\lib\optparse.py", line 1085, in format_help
    result.append(self.format_option_help(formatter))
  File "c:\python27\lib\optparse.py", line 1074, in format_option_help
    result.append(formatter.format_option(option))
  File "c:\python27\lib\optparse.py", line 316, in format_option
    help_text = self.expand_default(option)
  File "c:\python27\lib\site-packages\pip\baseparser.py", line 112, in expand_default
    return optparse.IndentedHelpFormatter.expand_default(self, option)
  File "c:\python27\lib\optparse.py", line 288, in expand_default
    return option.help.replace(self.default_tag, str(default_value))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdc' in position 9: ordinal not in range(128)

Interestingly installation and uninstallation seem to work fine also with this account. I guess the difference with the problem I saw earlier could be that my main user/admin doesn't have non-ASCII characters.

---

UPDATE: It later turned out that pip install is totally broken if the hostname has non-ASCII characters. That explains why creating a new account with just ASCII characters in the username didn't didn't work when I encountered this first time and also why I couldn't reproduce that more severe problem with just an account with non-ASCII username.

A workaround for both of these problems is using --no-cache-dir. Both problems are also fixed by PR #3970 that hopefully gets merged and released at some point.

[xavfernandez]

Hello, what is the pip version ? pip --version

[pekkaklarck]

On my machine I got 8.0.2. The other setup I saw had a fresh Python 2.7.11 installation. Don't remember pip version, but it must have been the one bundled with the Python 2.7.11 Windows installer.

[piotr-dobrogost]

In the past problem with non-ASCII usernames on Windows was raised in issue #1713. In that case the traceback was also pointing to optparse module.

[pekkaklarck]

Faced this again on a new training course. This time username had no non-ASCII characters, but Windows had Finnish locale and thus used C:\Käyttäjät\Name instead of C:\Users\Name. Still using Python 2.7.11 and installation was fresh.

This time our workaround was downloading the package and using pip install package-1.0.tar.gz. Less work than extracting the package and using python setup.py install, but required installing dependencies using the same approach first.

Would someone be interested to help me with the fix/PR if I try to debug why this actually happens? What timeline would we have to get the fix into Python 2.7.12? This is a very annoying bug and gives new users a bad first impression about pip and Python. It must be fixed in a version distributed with Python because the bug prevents upgrading pip itself.

[pfmoore]

Pure speculation, but that sounds like it might be where pip is trying to put the download cache. Maybe as a workaround you could try using the --cache-dir option to ask pip to put the cache somewhere without non-ASCII characters? Even if it's no better than your current workaround, if it works it might help pinpoint the issue.

[pekkaklarck]

Yes, --cache-dir seems to be causing this. I debugged the problem on my machine (see the original description) and noticed that it crashes when formatting help text for exactly that option. Looking at the pip and optparse code I think I found the root cause:

• pip sets default value for --cache-dir based on what pip.utils.appdirs.user_cache_dir returns. It is a Unicode string and in my case it is u'C:\\Users\\\xdcrj\xf6\\AppData\\Local\\pip\\Cache'.
• optparse formats the default value in HelpFormatter.expand_default (line 288) and uses str(default_value) there.

Based on this analysis the real bug would be in optparse. Not sure what's the best way to avoid it on pip side.

[pfmoore]

Yep, that analysis is exactly right. It's frustrating, because we don't even try to display the default value. It's also only an issue on Python 2 (of course... :-()

A bug report against cpython might be worth it. But it'd be nice to work around it in pip, as "upgrade to 2.7.12" seems like a bit of a heavy handed recommendation for this.

The only fix I can see within pip is to set the default for --cache-dir to None, and then after we've done option parsing, set the actual default if the user hasn't overridden it. But I'm not too familiar with pip's option system, so I don't know precisely where that change would need to go (and I don't have the time right now to dig into the code).

[pekkaklarck]

Just realized that it is likely that there are also other problems related to this. My students got errors when installing a package, not when showing help like I do, and I think their tracebacks were also different than what I get. Unfortunately I cannot reproduce that problem nor did I ask them to send me tracebacks. I can try creating a new Windows virtual machine with the main user having a non-ASCII user name and can also play with locale settings. We also continue the latest training next Monday and I can debug this in my student's machine then.

[pekkaklarck]

Yeah, this isn't just a problem with optparse. I installed new Windows 7 machine with main user Päivi, installed Python 2.7.11 and got the following traceback when running pip install robotframework.

Exception:
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\pip\basecommand.py", line 211, in main
    status = self.run(options, args)
  File "c:\python27\lib\site-packages\pip\commands\install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "c:\python27\lib\site-packages\pip\req\req_set.py", line 461, in _prepare_file
    req_to_install.populate_link(finder, self.upgrade)
  File "c:\python27\lib\site-packages\pip\req\req_install.py", line 250, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "c:\python27\lib\site-packages\pip\index.py", line 486, in find_requirement
    all_versions = self._find_all_versions(req.name)
  File "c:\python27\lib\site-packages\pip\index.py", line 404, in _find_all_versions
    index_locations = self._get_index_urls_locations(project_name)
  File "c:\python27\lib\site-packages\pip\index.py", line 378, in _get_index_urls_locations
    page = self._get_page(main_index_url)
  File "c:\python27\lib\site-packages\pip\index.py", line 818, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "c:\python27\lib\site-packages\pip\index.py", line 928, in get_page
    "Cache-Control": "max-age=600",
  File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "c:\python27\lib\site-packages\pip\download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\python27\lib\site-packages\pip\_vendor\requests\sessions.py", line 605, in send
    r.content
  File "c:\python27\lib\site-packages\pip\_vendor\requests\models.py", line 750, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "c:\python27\lib\site-packages\pip\_vendor\requests\models.py", line 673, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "c:\python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 307, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "c:\python27\lib\site-packages\pip\_vendor\requests\packages\urllib3\response.py", line 243, in read
    data = self._fp.read(amt)
  File "c:\python27\lib\site-packages\pip\_vendor\cachecontrol\filewrapper.py", line 54, in read
    self.__callback(self.__buf.getvalue())
  File "c:\python27\lib\site-packages\pip\_vendor\cachecontrol\controller.py", line 244, in cache_response
    self.serializer.dumps(request, response, body=body),
  File "c:\python27\lib\site-packages\pip\download.py", line 276, in set
    return super(SafeFileCache, self).set(*args, **kwargs)
  File "c:\python27\lib\site-packages\pip\_vendor\cachecontrol\caches\file_cache.py", line 99, in set
    with self.lock_class(name) as lock:
  File "c:\python27\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 18, in __init__
    LockBase.__init__(self, path, threaded, timeout)
  File "c:\python27\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 189, in __init__
    hash(self.path)))
  File "c:\python27\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 1: ordinal not in range(128)

[pekkaklarck]

Also tested that using --no-cache-dir works as a workaround.

[pfmoore]

OK, I suspect this is just symptomatic of the usual "Python 2.7 is rubbish at working with Unicode" situation. It's possible that changing _get_win_folder in pip.utils.appdirs to return a (byte) string under Python 2, encoded in a suitable encoding, would help, but I don't really know what encoding would be appropriate, nor do I know if that would simply introduce other problems.

This probably needs someone with Python 2/Unicode experience on Windows to look into it. I'm a Python 3 person myself, so I can't really offer much more. Sorry.

At least you have a workaround, which is something I guess...

[pekkaklarck]

Little more debugging. The problem occurs on lockfile/__init__.py, line 189, in this code:

        self.unique_name = os.path.join(dirname,
                                        "%s%s.%s%s" % (self.hostname,
                                                       self.tname,
                                                       self.pid,
                                                       hash(self.path)))

The reason is that dirname is a Unicode string (created based on cache dir) and self.hostname is a byte string returned by socket.gethostname() and both contain non-ASCII characters:

u'c:\\users\\p\xe4ivi\\appdata\\local\\pip\\cache\\http\\3\\f\\b\\1\\f'
'P\xe4ivi-PC'

This would explain why the problem doesn't occur in my main Windows virtual machine which has only ASCII characters in its host name. Unfortunately the default machine name Windows creates is based on the user name (like 'Päivi-PC' in my case).