remove the pyprocessing spkg from sage, then sort out any fallout that results

[williamstein]

In sage-4.1 we ship pyprocessing-0.52.p0.spkg, however a newer better version of that same code is standard in Python-2.6.x, which we also ship. So we need to remove pyprocessing, do a clean build/test cycle, and see what problems arise, then sort them out.

Be sure to test "sage -tp", which isn't tested by the test suite unless you do "make ptest".

CC: @mwhansen

Component: packages: standard

Author: John Palmieri, Mike Hansen

Reviewer: Minh Van Nguyen

Merged: sage-4.3.4.alpha0

Issue created by migration from https://trac.sagemath.org/ticket/6503

[williamstein]

comment:1

Just delete the spkg and delete the relevant lines in spkg/standard/deps. That should be it.

[sagetrac-mvngu]

comment:3

This might not be as easy as it seems. In my merge tree, I tried removing the pyprocessing spkg, and edited spkg/install and spkg/standard/deps to get rid of the rules relating to building pyprocessing. Based on this tree with pyprocessing removed, I cut a source tarball and parallel compile (using "export MAKE='make -j4'") everything from source. Trouble kicks in while compiling the Sage library. Here's a relevant snippet from the install log:

Building modified file sage/ext/interpreters/wrapper_el.pyx.
Execute 248 commands (using 4 threads)
Traceback (most recent call last):
  File "setup.py", line 752, in <module>
    execute_list_of_commands(queue)
  File "setup.py", line 249, in execute_list_of_commands
    execute_list_of_commands_in_parallel(command_list, nthreads)
  File "setup.py", line 188, in execute_list_of_commands_in_parallel
    from processing import Pool
ImportError: No module named processing
sage: There was an error installing modified sage library code.


real    0m3.261s
user    0m2.540s
sys     0m0.770s
Error building new version of SAGE.
You might try typing 'sage -ba' or write to sage-support with as much information as possible.

real    0m7.376s
user    0m5.570s
sys     0m1.820s
sage: An error occurred while installing sage-4.3.1.alpha0-take1

The build error cannot be due to the patches or updated spkg's in my merge tree. I also tried cutting a source tarball of my merge tree (with pyprocessing intact where it is), and compile everything from source. This went OK. Removing pyrocessing would need to wait after releasing Sage 4.3.2.alpha0. I'm putting this back to "needs work". The likely place to look is the file devel/sage-main/setup.py, especially the following function:

def execute_list_of_commands_in_parallel(command_list, nthreads):
    """                                                                                                                                                                                                                            
    INPUT:                                                                                                                                                                                                                         
        command_list -- a list of pairs, consisting of a                                                                                                                                                                           
             function to call and its argument                                                                                                                                                                                     
        nthreads -- integer; number of threads to use                                                                                                                                                                              
                                                                                                                                                                                                                                   
    OUTPUT:                                                                                                                                                                                                                        
        Executes the given list of commands, possibly in parallel,                                                                                                                                                                 
        using nthreads threads.  Terminates setup.py with an exit code of 1                                                                                                                                                        
        if an error occurs in any subcommand.                                                                                                                                                                                      
                                                                                                                                                                                                                                   
    WARNING: commands are run roughly in order, but of course successive                                                                                                                                                           
    commands may be run at the same time.                                                                                                                                                                                          
    """
    print "Execute %s commands (using %s threads)"%(len(command_list), min(len(command_list),nthreads))
    from processing import Pool
    p = Pool(nthreads)
    for r in p.imap(apply_pair, command_list):
        if r:
            print "Parallel build failed with status %s."%r
            sys.exit(1)

[maxthemouse]

comment:4

According to: http://www.python.org/dev/peps/pep-0371/ there was a renaming from processing to multiprocessing. There may be other changes to the api as well.

Adam