CalledProcessError attempting to add a blurb

[jaraco]

I'm attempting to use blurb for the first time, but right off the bat, I'm getting a CalledProcessError:

cpython issue-34108 $ rwt blurb -- -m blurb add
Collecting blurb
  Downloading https://files.pythonhosted.org/packages/41/90/cdce96c5f5767e08a87dad88cb2da0b962c692e826d2083d983d1f6f4019/blurb-1.0.7-py3-none-any.whl (62kB)
    100% |████████████████████████████████| 71kB 374kB/s
Installing collected packages: blurb
Successfully installed blurb-1.0.7
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-f3yblb6y/blurb.py", line 1664, in <module>
    main()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-f3yblb6y/blurb.py", line 1624, in main
    sys.exit(fn(*filtered_args, **kwargs))
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-f3yblb6y/blurb.py", line 952, in add
    flush_git_add_files()
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-f3yblb6y/blurb.py", line 1111, in flush_git_add_files
    subprocess.run(["git", "add", "-f", *git_add_files], stdout=subprocess.PIPE, stderr=subprocess.PIPE).check_returncode()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 409, in check_returncode
    self.stderr)
subprocess.CalledProcessError: Command '['git', 'add', '-f', '/Users/jaraco/Dropbox/code/public/cpython/Misc/NEWS.d/next/Library/2018-07-13-08-44-52.bpo-34108.RjobUC.rst']' returned non-zero exit status 128.

[jaraco]

Running the command manually succeeded. Not sure why git would fail with an exit code of 128 and not emit any output. Oh, because the code swallows the output.

[jaraco]

Running a second time with pdb, I did not encounter the error (though I did encounter another error):

cpython bdd249c68f $ rwt blurb -- -m pdb -m blurb
Collecting blurb
  Using cached https://files.pythonhosted.org/packages/41/90/cdce96c5f5767e08a87dad88cb2da0b962c692e826d2083d983d1f6f4019/blurb-1.0.7-py3-none-any.whl
Installing collected packages: blurb
Successfully installed blurb-1.0.7
> /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py(2)<module>()
-> """Command-line tool to manage CPython Misc/NEWS.d entries."""
(Pdb) b blurb:1111
Breakpoint 1 at /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py:1111
(Pdb) c
> /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py(1111)flush_git_add_files()
-> subprocess.run(["git", "add", "-f", *git_add_files], stdout=subprocess.PIPE, stderr=subprocess.PIPE).check_returncode()
(Pdb) subprocess.PIPE=None
(Pdb) n
> /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py(1112)flush_git_add_files()
-> git_add_files.clear()
(Pdb) c
Ready for commit.
The program exited via sys.exit(). Exit status:
> /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py(2)<module>()
-> """Command-line tool to manage CPython Misc/NEWS.d entries."""
(Pdb) q
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/rwt-1gtoodu2/blurb.py", line 888, in <lambda>
    atexit.register(lambda : os.unlink(tmp_path))
NameError: name 'os' is not defined

[brettcannon]

@jaraco how is git put on your PATH? It looks like shell=True isn't set which would mean that if you're adding git through e.g. .bashrc it wouldn't be available to subprocess.

[jaraco]

I'm on macOS 10.13.6 with git 2.18.0 in /usr/local/bin as installed by homebrew. And I use programs dozens of times daily that invoke subprocess.Popen(['git', ...]) (manly setuptools_scm).

I honestly can't imagine what might have gone wrong.

I tried running the command a couple more times (same as before), but this time didn't encounter the error.

Still, I wonder if it would be better to let the output from the git invocation be piped directly to stdout/stderr:

core-workflow master $ git diff
diff --git a/blurb/blurb.py b/blurb/blurb.py
index 10dac4d..001c7bd 100755
--- a/blurb/blurb.py
+++ b/blurb/blurb.py
@@ -697,7 +697,7 @@ def chdir_to_repo_root():
     # find the root of the local CPython repo
     # note that we can't ask git, because we might
     # be in an exported directory tree!
-
+
     # we intentionally start in a (probably nonexistant) subtree
     # the first thing the while loop does is .., basically
     path = os.path.abspath("garglemox")
@@ -1108,14 +1108,14 @@ Python News
 git_add_files = []
 def flush_git_add_files():
     if git_add_files:
-        subprocess.run(["git", "add", "-f", *git_add_files], stdout=subprocess.PIPE, stderr=subprocess.PIPE).check_returncode()
+        subprocess.run(["git", "add", "-f", *git_add_files]).check_returncode()
         git_add_files.clear()

 git_rm_files = []
 def flush_git_rm_files():
     if git_rm_files:
         try:
-            subprocess.run(["git", "rm", "-f", *git_rm_files], stdout=subprocess.PIPE, stderr=subprocess.PIPE).check_returncode()
+            subprocess.run(["git", "rm", "-f", *git_rm_files]).check_returncode()
         except subprocess.CalledProcessError:
             pass

Or if there's some reason to capture the output (there isn't any output at all in my use case), use that output when reporting an exception.

[jaraco]

I can see there is some output from git rm that would be suppressed by the PIPE parameter. Perhaps better would be to use git rm -q if blurb wishes for the command to run with minimal output.

[webknjaz]

JFYI $ git rm -f non-existing-file returns code 128. This could've happened because of double-run of this command on the same file, for example. (First run removes file from fs successfully with rc 0, second fails to locate it and returns 128)

[webknjaz]

Also, this happens to untracked files (not git added):

$ touch some_file
$ git rm -f some_file
fatal: pathspec 'some_file' did not match any files
$ echo $?
128

[jaraco]

The original error I hit was during git add.

[webknjaz]

git adding non-existing file is also 128.