Upgrade to support Slurm 20.02

[bhipple]

https://github.com/SchedMD/slurm/releases
Upstream Slurm has had a 20.02 release and we're interested in upgrading to it in NixPkgs. What's involved to get pyslurm to support the new release?

[giovtorres]

My test container is broken between 19.05 and 20.02 and I haven't had a chance to troubleshoot 😭

[laurentvm]

Naive attempt to change only SLURM_VERSION to "20.02

pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_6config___get’:
pyslurm/pyslurm.c:12144:75: error: ‘slurm_ctl_conf_t’ has no member named ‘acctng_store_job_comment’
     __pyx_t_1 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->acctng_store_job_comment); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 518, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c:12288:86: error: ‘slurm_ctl_conf_t’ has no member named ‘checkpoint_type’
     __pyx_t_1 = __pyx_f_7pyslurm_5slurm_stringOrNone(__pyx_v_self->__pyx___Config_ptr->checkpoint_type, __pyx_kp_s_); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 530, __pyx_L1_error)
                                                                                      ^
pyslurm/pyslurm.c:12420:75: error: ‘slurm_ctl_conf_t’ has no member named ‘disable_root_jobs’
     __pyx_t_1 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->disable_root_jobs); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 541, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c:12510:75: error: ‘slurm_ctl_conf_t’ has no member named ‘fast_schedule’
     __pyx_t_1 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->fast_schedule); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 548, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c:12717:86: error: ‘slurm_ctl_conf_t’ has no member named ‘job_ckpt_dir’
     __pyx_t_2 = __pyx_f_7pyslurm_5slurm_stringOrNone(__pyx_v_self->__pyx___Config_ptr->job_ckpt_dir, __pyx_kp_s_); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 565, __pyx_L1_error)
                                                                                      ^
pyslurm/pyslurm.c:12954:90: error: ‘slurm_ctl_conf_t’ has no member named ‘licenses_used’
     __pyx_t_1 = __pyx_f_7pyslurm_7pyslurm___get_licenses(__pyx_v_self->__pyx___Config_ptr->licenses_used); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 591, __pyx_L1_error)
                                                                                          ^
pyslurm/pyslurm.c:14390:75: error: ‘slurm_ctl_conf_t’ has no member named ‘track_wckey’
     __pyx_t_1 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->track_wckey); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 716, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c:14438:75: error: ‘slurm_ctl_conf_t’ has no member named ‘use_pam’
     __pyx_t_1 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->use_pam); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 720, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c:14453:75: error: ‘slurm_ctl_conf_t’ has no member named ‘use_spec_resources’
     __pyx_t_2 = __Pyx_PyInt_From_uint16_t(__pyx_v_self->__pyx___Config_ptr->use_spec_resources); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 721, __pyx_L1_error)
                                                                           ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_able’:
pyslurm/pyslurm.c:23843:3: warning: implicit declaration of function ‘slurm_checkpoint_able’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_able(__pyx_v_JobID, __pyx_v_JobStep, (&__pyx_v_Time));
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_enable’:
pyslurm/pyslurm.c:24080:3: warning: implicit declaration of function ‘slurm_checkpoint_enable’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_enable(__pyx_v_JobID, __pyx_v_JobStep);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_disable’:
pyslurm/pyslurm.c:24317:3: warning: implicit declaration of function ‘slurm_checkpoint_disable’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_disable(__pyx_v_JobID, __pyx_v_JobStep);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_create’:
pyslurm/pyslurm.c:24562:3: warning: implicit declaration of function ‘slurm_checkpoint_create’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_create(__pyx_v_JobID, __pyx_v_JobStep, __pyx_v_MaxWait, __pyx_v_ImageDir);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_requeue’:
pyslurm/pyslurm.c:24837:3: warning: implicit declaration of function ‘slurm_checkpoint_requeue’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_requeue(__pyx_v_JobID, __pyx_v_MaxWait, __pyx_v_ImageDir);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_restart’:
pyslurm/pyslurm.c:25386:3: warning: implicit declaration of function ‘slurm_checkpoint_restart’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_restart(__pyx_v_JobID, __pyx_v_JobStep, __pyx_v_Stick, __pyx_v_ImageDir);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_complete’:
pyslurm/pyslurm.c:25669:3: warning: implicit declaration of function ‘slurm_checkpoint_complete’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_complete(__pyx_v_JobID, __pyx_v_JobStep, __pyx_v_BeginTime, __pyx_v_ErrorCode, __pyx_v_ErrMsg);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_task_complete’:
pyslurm/pyslurm.c:25973:3: warning: implicit declaration of function ‘slurm_checkpoint_task_complete’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_task_complete(__pyx_v_JobID, __pyx_v_JobStep, __pyx_v_TaskID, __pyx_v_BeginTime, __pyx_v_ErrorCode, __pyx_v_ErrMsg);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_pf_7pyslurm_7pyslurm_74slurm_checkpoint_error’:
pyslurm/pyslurm.c:26357:3: warning: implicit declaration of function ‘slurm_checkpoint_error’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_error(__pyx_v_JobID, __pyx_v_JobStep, (&__pyx_v_ErrorCode), (&__pyx_v_Msg));
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_slurm_checkpoint_tasks’:
pyslurm/pyslurm.c:26556:3: warning: implicit declaration of function ‘slurm_checkpoint_tasks’ [-Wimplicit-function-declaration]
   __pyx_v_errCode = slurm_checkpoint_tasks(__pyx_v_JobID, __pyx_v_JobStep, __pyx_v_BeginTime, __pyx_v_ImageDir, __pyx_v_MaxWait, __pyx_v_NodeList);
   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_3job_get_job_ptr’:
pyslurm/pyslurm.c:29906:39: error: ‘slurm_job_info_t’ has no member named ‘pack_job_id’
     __pyx_t_6 = (__pyx_v_self->_record->pack_job_id != 0);
                                       ^
pyslurm/pyslurm.c:29916:66: error: ‘slurm_job_info_t’ has no member named ‘pack_job_id’
       __pyx_t_8 = __Pyx_PyInt_From_uint32_t(__pyx_v_self->_record->pack_job_id); if (unlikely(!__pyx_t_8)) __PYX_ERR(0, 2216, __pyx_L1_error)
                                                                  ^
pyslurm/pyslurm.c:29928:66: error: ‘slurm_job_info_t’ has no member named ‘pack_job_offset’
       __pyx_t_8 = __Pyx_PyInt_From_uint32_t(__pyx_v_self->_record->pack_job_offset); if (unlikely(!__pyx_t_8)) __PYX_ERR(0, 2217, __pyx_L1_error)
                                                                  ^
pyslurm/pyslurm.c:29949:39: error: ‘slurm_job_info_t’ has no member named ‘pack_job_id_set’
     __pyx_t_6 = (__pyx_v_self->_record->pack_job_id_set != 0);
                                       ^
pyslurm/pyslurm.c:29959:65: error: ‘slurm_job_info_t’ has no member named ‘pack_job_id_set’
       __pyx_t_8 = __Pyx_PyBytes_FromString(__pyx_v_self->_record->pack_job_id_set); if (unlikely(!__pyx_t_8)) __PYX_ERR(0, 2220, __pyx_L1_error)
                                                                 ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_3job_fill_job_desc_from_opts’:
pyslurm/pyslurm.c:36875:15: error: ‘job_desc_msg_t’ has no member named ‘ckpt_dir’
   __pyx_v_desc->ckpt_dir = slurm_get_checkpoint_dir();
               ^
pyslurm/pyslurm.c:36909:17: error: ‘job_desc_msg_t’ has no member named ‘ckpt_interval’
     __pyx_v_desc->ckpt_interval = ((uint16_t)__pyx_t_4);
                 ^
pyslurm/pyslurm.c:36929:17: error: ‘job_desc_msg_t’ has no member named ‘ckpt_interval’
     __pyx_v_desc->ckpt_interval = ((uint16_t)0);
                 ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_7jobstep___get’:
pyslurm/pyslurm.c:47339:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (__pyx_t_8 = 0; __pyx_t_8 < __pyx_t_7; __pyx_t_8+=1) {
                                   ^
pyslurm/pyslurm.c:47888:106: error: ‘job_step_info_t’ has no member named ‘ckpt_dir’
       __pyx_t_12 = __pyx_f_7pyslurm_5slurm_stringOrNone((__pyx_v_job_step_info_ptr->job_steps[__pyx_v_i]).ckpt_dir, __pyx_kp_s_); if (unlikely(!__pyx_t_12)) __PYX_ERR(0, 
                                                                                                          ^
pyslurm/pyslurm.c:47900:95: error: ‘job_step_info_t’ has no member named ‘ckpt_interval’
       __pyx_t_12 = __Pyx_PyInt_From_uint16_t((__pyx_v_job_step_info_ptr->job_steps[__pyx_v_i]).ckpt_interval); if (unlikely(!__pyx_t_12)) __PYX_ERR(0, 3738, __pyx_L1_erro
                                                                                               ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_8topology___get’:
pyslurm/pyslurm.c:58406:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     for (__pyx_t_8 = 0; __pyx_t_8 < __pyx_t_7; __pyx_t_8+=1) {

[bikerdanny]

I forked the repo (https://github.com/bikerdanny/pyslurm) and did some further changes.
It would be nice if the upgrade process was documented.
Then one could help better.

[giovtorres]

I have a minimal set of notes in the wiki: https://github.com/PySlurm/pyslurm/wiki/Updating-PySlurm-for-New-Slurm-Releases. The git diff commands show the differences between the major versions.

All the C structs have to be updated between major versions. You can also look at previous commits that upgrade major releases.

I appreciate any contributions.

[bikerdanny]

Thank you very much!

[mick-t]

What's the best way to help? Create branch of the master branch or off the 19.05.0-v2 branch?

[giovtorres]

Hi @mick-t , for starters, I need Slurm 20.02.0 working inside a test container. This is how I test locally. See this issue: https://groups.google.com/d/topic/slurm-users/GW6P5bHIpGQ/discussion

I have something wrong with my config. I've been using this config for several releases, but it's not working with the latest Slurm.

Have you gotten Slurm working inside a VM or container?

[giovtorres]

With respect to the branches, the easier of the two is to branch off 19.05.0. It is currently in sync with master.

I'd really like to get v2 of the API going. I started but could never finish before a new version of Slurm released. v2 is faster and more Pythonic overall, using objects instead of dictionaries to access values.

If I can get Slurm 20.02.0 working locally, I will likely try to get v2 going again for the current version of Slurm and eventually deprecate v1 after the next release.

I'd appreciate any help with the project.

[mick-t]

Hi Giovanni,

I'm not yet running Slurm 20.x and have just starting the process to upgrade my instance at work. I may not have a running instance to test for a while.

Can I get a copy of your container or container definition? I can see if I can run it and figure out your issues.

[giovtorres]

Can I get a copy of your container or container definition? I can see if I can run it and figure out your issues.

https://github.com/giovtorres/docker-centos7-slurm

[bikerdanny]

Hello Giovanni,

I checked out your docker-centos7-slurm and build successully the new Slurm version 20.02.1.
docker build --build-arg SLURM_TAG="slurm-20-02-1-1" -t docker-centos7-slurm:20.02.1-1 .
docker run -it --name ernie --hostname ernie docker-centos7-slurm:20.02.1-1 bash

• Initializing database
• Updating MySQL directory permissions
• Starting MariaDB to create Slurm account database
  200408 10:41:43 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
  200408 10:41:43 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
• Starting MariaDB to create Slurm account database
• Starting MariaDB to create Slurm account database
• Creating Slurm acct database
• Slurm acct database created. Stopping MariaDB
• Starting all Slurm processes under supervisord
  [root@ernie /]# sinfo
  PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
  normal* up 5-00:00:00 5 idle c[1-5]
  debug up 5-00:00:00 5 down c[6-10]
  [root@ernie /]# srun hostname
  ernie

If I use the parameter "-d" instead of "-it", than the container exited after your entrypoint script, because bash exit immediately.
I use the following in my docker container.
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["supervisord", "-c", "/etc/supervisord.conf", "--nodaemon"]

Best wishes,
Danny

[bikerdanny]

...ok, I want to correct myself, as you wrote correctly in your docker-compose.yml, only a terminal (-t) is needed, so the following command also works for me.
docker run -dt --name ernie --hostname ernie docker-centos7-slurm:20.02.1-1

[bikerdanny]

Do you use tools to generate the pxd file automatically?
https://github.com/cython/cython/wiki/AutoPxd