Function interface crash when upgrading nipype

[MariaICL]

Summary

The function interface crashes, in a code that was working in an older nipype installation (version details below). From the error, it seems I am doing something wrong when importing other functions inside of the function interface, but I don't know what it is exactly or how to fix it.

The function itself is intended to convert html into pdf and send it as an email attachment.

Actual behavior

The function below does not run, and this is the crash information:

170601-20:23:45,389 workflow INFO:
	 Traceback (most recent call last):
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 613, in _run_command
    result = self._interface.run()
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1081, in run
    runtime = self._run_wrapper(runtime)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1029, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/utility/wrappers.py", line 146, in _run_interface
    self.imports)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/utils/misc.py", line 102, in create_function_from_source
    raise_from(RuntimeError(msg), e)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/future/utils/__init__.py", line 454, in raise_from
    raise e
RuntimeError: 
Error executing function:
 
Functions in connection strings have to be standalone.
They cannot be declared either interactively or inside
another function or inline in the connect string. Any
imports should be done inside the function
Interface Function failed to run. 

170601-20:23:46,403 workflow INFO:
Traceback (most recent call last):
	 ***********************************
  File "/home/myanezl2/workspace/c3nl/QA_matlab/QA_Matlab.py", line 192, in <module>
170601-20:23:46,403 workflow ERROR:
	 could not run node: QA.QA_email
170601-20:23:46,404 workflow INFO:
	 crashfile: /home/myanezl2/workspace/c3nl/QA_matlab/crash-20170601-202344-myanezl2-QA_email-98e03d45-765c-4bec-87c5-c4e253113c83.pklz
170601-20:23:46,404 workflow INFO:
	 ***********************************
    QA.run('MultiProc', plugin_args={'n_procs': 2})
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.py", line 590, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 279, in run
    report_nodes_not_run(notrun)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 101, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details

Process finished with exit code 1

Expected behavior

This function was working in this previous installation:
print(nipype.version)
0.11.0

print(nipype.get_info())
{'nibabel_version': '2.0.2', 'networkx_version': '1.10', 'numpy_version': '1.11.0', 'sys_platform': 'linux2', 'sys_version': '2.7.11 |Anaconda 2.4.1 (64-bit)| (default, Dec 6 2015, 18:08:32) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]', 'commit_source': '(none found)', 'commit_hash': '', 'pkg_path': '/share/apps/anaconda-3-2.4.1/lib/python2.7/site-packages/nipype', 'sys_executable': '/share/apps/anaconda-3-2.4.1/bin/python', 'traits_version': '4.5.0', 'scipy_version': '0.17.0'}

Script/Workflow details

def gen_qa_email(in_directory, empty_variable, in_date):
	import os
	import smtplib
	import pdfkit
	from email.mime.application import MIMEApplication
	from email.mime.multipart import MIMEMultipart
	from email.mime.text import MIMEText
	from email.mime.image import MIMEImage

	""" Convert html to pdf"""
	pdf_file = pdfkit.from_file(in_directory + '/QA_results.html', False)

	""" Create the email message container from the input args."""
	recipients = ['email@email.com']
	msg = MIMEMultipart('alternative')
	msg['Subject'] = 'QA' + in_date
	msg['From'] = 'email@email.com'
	msg['To'] = ", ".join(recipients)

	""" Attach parts into message container"""
	pdf = MIMEApplication(pdf_file, 'pdf')
	pdf.add_header('Content-Disposition', 'attachment', filename= 'QA.pdf')
	msg.attach(pdf)

	try:
		smtpserver = smtplib.SMTP("smtp.gmail.com", 587)
		smtpserver.ehlo()
		smtpserver.starttls()
		smtpserver.ehlo
		smtpserver.login(''email@email.com']', 'password')
		smtpserver.sendmail(''email@email.com'], recipients, msg.as_string())
		smtpserver.quit()
	except smtplib.SMTPAuthenticationError as e:
		print "Unable to send message: %s" % e
	return None

QA_email = Node(name='QA_email', interface=Function(input_names=['in_directory', 'empty_variable', 'in_date'], output_names=[], function=gen_qa_email))
QA_email.inputs.in_date = experiment_date

Platform details:

print(nipype.version)
0.13.1

print(nipype.get_info())
{'nibabel_version': '2.1.0', 'sys_executable': '/apps/software/anaconda/4.3.1/bin/python', 'networkx_version': '1.11', 'numpy_version': '1.11.3', 'sys_platform': 'linux2', 'sys_version': '2.7.13 |Anaconda 4.3.1 (64-bit)| (default, Dec 20 2016, 23:09:15) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]', 'commit_source': u'installation', 'commit_hash': u'8946bcab9', 'pkg_path': '/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype', 'nipype_version': u'0.13.1', 'traits_version': '4.6.0', 'scipy_version': '0.18.1'}

[mgxd]

@MariaICL could you try removing the imports from inside the function and instead assigning them to the Function interface? Something like:

qa_email_imports = ['import os',
	'import smtplib',
	'import pdfkit',
	'from email.mime.application import MIMEApplication',
	'from email.mime.multipart import MIMEMultipart',
	'from email.mime.text import MIMEText',
	'from email.mime.image import MIMEImage']

QA_email = Node(name='QA_email', interface=Function(input_names=['in_directory', 'empty_variable', 'in_date'], output_names=[], function=gen_qa_email, imports=qa_email_imports))

[MariaICL]

Thanks for the tip, but I just tried it and I got exactly the same error...

Executing node QA_email in dir: /home/myanezl2/test_nipype_3/expdir_QA/2017-05-22/QA/QA_email
170602-17:11:35,952 workflow ERROR:
	 [u'Node QA_email failed to run on host brodmann23d.c3nl.']
170602-17:11:36,73 workflow INFO:
	 Saving crash info to /home/myanezl2/workspace/c3nl/QA_matlab/crash-20170602-171135-myanezl2-QA_email-4cd823f1-c6b0-495d-83f6-fa28039191e5.pklz
170602-17:11:36,73 workflow INFO:
	 Traceback (most recent call last):
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 372, in run
    self._run_interface()
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 482, in _run_interface
    self._result = self._run_command(execute)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 613, in _run_command
    result = self._interface.run()
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1081, in run
    runtime = self._run_wrapper(runtime)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1029, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/interfaces/utility/wrappers.py", line 146, in _run_interface
    self.imports)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/utils/misc.py", line 102, in create_function_from_source
    raise_from(RuntimeError(msg), e)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/future/utils/__init__.py", line 454, in raise_from
    raise e
RuntimeError: 
Error executing function:
 def gen_qa_email(in_directory, empty_variable, in_date):

	""" Convert html to pdf"""
	pdf_file = pdfkit.from_file(in_directory + '/QA_results.html', False)

	""" Create the email message container from the input args."""
	recipients = ['email@email.com', 'email@email.com']
	msg = MIMEMultipart('alternative')
	msg['Subject'] = 'QA fmri and dti ' + in_date
	msg['From'] = 'email@email.com'
	msg['To'] = ", ".join(recipients)

	""" Attach parts into message container"""
	pdf = MIMEApplication(pdf_file, 'pdf')
	pdf.add_header('Content-Disposition', 'attachment', filename= 'QA_fMRI.pdf')
	msg.attach(pdf)

	try:
		smtpserver = smtplib.SMTP("smtp.gmail.com", 587)
		smtpserver.ehlo()
		smtpserver.starttls()
		smtpserver.ehlo
		smtpserver.login('email@email.com', 'password')
		smtpserver.sendmail('email@email.com', recipients, msg.as_string())
		smtpserver.quit()
	except smtplib.SMTPAuthenticationError as e:
		print "Unable to send message: %s" % e
	return None

Functions in connection strings have to be standalone.
They cannot be declared either interactively or inside
another function or inline in the connect string. Any
imports should be done inside the function
Interface Function failed to run. 

Traceback (most recent call last):
  File "/home/myanezl2/workspace/c3nl/QA_matlab/QA_Matlab.py", line 193, in <module>
170602-17:11:36,602 workflow INFO:
	 ***********************************
170602-17:11:36,602 workflow ERROR:
	 could not run node: QA.QA_email
170602-17:11:36,602 workflow INFO:
	 crashfile: /home/myanezl2/workspace/c3nl/QA_matlab/crash-20170602-171135-myanezl2-QA_email-4cd823f1-c6b0-495d-83f6-fa28039191e5.pklz
170602-17:11:36,602 workflow INFO:
	 ***********************************
    QA.run('MultiProc', plugin_args={'n_procs': 2})
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.py", line 590, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 279, in run
    report_nodes_not_run(notrun)
  File "/apps/software/anaconda/4.3.1/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 101, in report_nodes_not_run
    raise RuntimeError(('Workflow did not execute cleanly. '
RuntimeError: Workflow did not execute cleanly. Check log for details

Process finished with exit code 1

[mgxd]

Is empty_variable even used? that may be the problem with your function.

[MariaICL]

I took empty_variable out, it was not being used, but the error is still the same...

[mgxd]

@MariaICL I think the problem here is this line:

print "Unable to send message: %s" % e

Since Nipype has moved to primarily supporting Python 3, try replacing that with:

print("Unable to send message: {}".format(e))

[MariaICL]

No difference in the error with that change, I am afraid...

[sgiavasis]

Noting here that I am also seeing this error when I use 0.13+ versions of Nipype- my function nodes start throwing this error, even if I try assigning the list of imports to the function node as recommended above and in #1097.

[effigies]

Can you give a minimal example of a function that fails?

[sgiavasis]

The function node declaration (rest of workflow cut out for brevity):

gather_imports = ['import os',
                  'import nibabel as nb']

gather_header = pe.Node(niu.Function(
                         input_names=['in_file', 'subject', 'session',
                                      'scan', 'type'],
                         output_names=['qap_dict'],
                         function=create_header_dict_entry,
                         imports=gather_imports),
                        name="gather_header_info_%s%s" % (data_type,
                                                          name))
gather_header.inputs.subject = config["subject_id"]
gather_header.inputs.session = config["session_id"]
gather_header.inputs.scan = config["scan_id"]

The function the function node is trying to run:

def create_header_dict_entry(in_file, subject, session, scan, type):
"""Gather the header information from a NIFTI file and arrange it into a
Python dictionary.

:type in_file: str
:param in_file: Filepath to the NIFTI raw data scan.
:type subject: str
:param subject: The participant ID.
:type session: str
:param session: The session ID.
:type scan: str
:param scan: The scan ID.
:type type: str
:param type: The data type ("anatomical" or "functional").
:rtype: dict
:return: A dictionary with the header information of the file, keyed by
         the participant's ID data.
"""

if not os.path.isfile(in_file):
    err = "Filepath doesn't exist!\nFilepath: %s" % in_file
    raise Exception(err)

try:
    img = nb.load(in_file)
    img_header = img.header
except:
    err = "You may not have an up-to-date installation of the Python " \
          "Nibabel package.\nYour Nibabel version: %s" % \
          str(nb.__version__)
    raise Exception(err)

subkey = "%s_header_info" % type
id_string = "%s %s %s" % (subject, session, scan)
qap_dict = {id_string: {subkey: {}}}

info_labels = ["descrip", "db_name", "bitpix", "slice_start",
               "scl_slope", "scl_inter", "slice_end", "slice_duration",
               "toffset", "quatern_b", "quatern_c", "quatern_d",
               "qoffset_x", "qoffset_y", "qoffset_z", "aux_file",
               "intent_name", "slice_code", "data_type", "qform_code",
               "srow_x", "srow_y", "srow_z", "sform_code"]

for info_label in info_labels:
    try:
        qap_dict[id_string][subkey][info_label] = \
            str(img_header[info_label])
    except:
        print "\n\n%s field not in NIFTI header of %s\n\n" % \
              (info_label, in_file)
        qap_dict[id_string][subkey][info_label] = ""
        pass

try:
    pixdim = img_header['pixdim']
    qap_dict[id_string][subkey]["pix_dimx"] = str(pixdim[1])
    qap_dict[id_string][subkey]["pix_dimy"] = str(pixdim[2])
    qap_dict[id_string][subkey]["pix_dimz"] = str(pixdim[3])
    qap_dict[id_string][subkey]["tr"] = str(pixdim[4])
except:
    print "\n\npix_dim/TR fields not in NIFTI header of %s\n\n" % in_file
    pass

try:
    qap_dict[id_string][subkey]["extensions"] = \
        len(img.header.extensions.get_codes())
except:
    print "\n\nExtensions not in NIFTI header of %s\n\n" % in_file
    pass

return qap_dict

The error:

[u'Node gather_header_info_func_site_UM1_sub_0050333_ses_1_task_rest_run_1_bold failed to run on host CMI-RSCH-MBP027.local.']
170622-16:53:42,530 workflow INFO:
Saving crash info to output/crash-20170622-165342-steven.giavasis-gather_header_info_func_site_UM1_sub_0050333_ses_1_task_rest_run_1_bold-0b9aa777-7d86-4d9a-9188-3e3ef1dc2891.pklz
170622-16:53:42,530 workflow INFO:
Traceback (most recent call last):
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/pipeline/plugins/multiproc.py", line 52, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/pipeline/engine/nodes.py", line 367, in run
self._run_interface()
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/pipeline/engine/nodes.py", line 477, in _run_interface
self._result = self._run_command(execute)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/pipeline/engine/nodes.py", line 607, in _run_command
result = self._interface.run()
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/interfaces/base.py", line 1085, in run
runtime = self._run_wrapper(runtime)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/interfaces/base.py", line 1033, in _run_wrapper
runtime = self._run_interface(runtime)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/interfaces/utility.py", line 450, in _run_interface
self.imports)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/nipype-0.13.0_ge1d8ec91a.dev-py2.7.egg/nipype/utils/misc.py", line 102, in create_function_from_source
raise_from(RuntimeError(msg), e)
File "/Users/steven.giavasis/miniconda2/lib/python2.7/site-packages/future/utils/init.py", line 454, in raise_from
raise e
RuntimeError:
Error executing function:
def create_header_dict_entry(in_file, subject, session, scan, type):
"""Gather the header information from a NIFTI file and arrange it into a
Python dictionary.

:type in_file: str
:param in_file: Filepath to the NIFTI raw data scan.
:type subject: str
:param subject: The participant ID.
:type session: str
:param session: The session ID.
:type scan: str
:param scan: The scan ID.
:type type: str
:param type: The data type ("anatomical" or "functional").
:rtype: dict
:return: A dictionary with the header information of the file, keyed by
         the participant's ID data.
"""

if not os.path.isfile(in_file):
    err = "Filepath doesn't exist!\nFilepath: %s" % in_file
    raise Exception(err)

try:
    img = nb.load(in_file)
    img_header = img.header
except:
    err = "You may not have an up-to-date installation of the Python " \
          "Nibabel package.\nYour Nibabel version: %s" % \
          str(nb.__version__)
    raise Exception(err)

subkey = "%s_header_info" % type
id_string = "%s %s %s" % (subject, session, scan)
qap_dict = {id_string: {subkey: {}}}

info_labels = ["descrip", "db_name", "bitpix", "slice_start",
               "scl_slope", "scl_inter", "slice_end", "slice_duration",
               "toffset", "quatern_b", "quatern_c", "quatern_d",
               "qoffset_x", "qoffset_y", "qoffset_z", "aux_file",
               "intent_name", "slice_code", "data_type", "qform_code",
               "srow_x", "srow_y", "srow_z", "sform_code"]

for info_label in info_labels:
    try:
        qap_dict[id_string][subkey][info_label] = \
            str(img_header[info_label])
    except:
        print "\n\n%s field not in NIFTI header of %s\n\n" % \
              (info_label, in_file)
        qap_dict[id_string][subkey][info_label] = ""
        pass

try:
    pixdim = img_header['pixdim']
    qap_dict[id_string][subkey]["pix_dimx"] = str(pixdim[1])
    qap_dict[id_string][subkey]["pix_dimy"] = str(pixdim[2])
    qap_dict[id_string][subkey]["pix_dimz"] = str(pixdim[3])
    qap_dict[id_string][subkey]["tr"] = str(pixdim[4])
except:
    print "\n\npix_dim/TR fields not in NIFTI header of %s\n\n" % in_file
    pass

try:
    qap_dict[id_string][subkey]["extensions"] = \
        len(img.header.extensions.get_codes())
except:
    print "\n\nExtensions not in NIFTI header of %s\n\n" % in_file
    pass

return qap_dict

Functions in connection strings have to be standalone.
They cannot be declared either interactively or inside
another function or inline in the connect string. Any
imports should be done inside the function
Interface Function failed to run.

[effigies]

Sorry, by "minimal example", I meant the smallest possible (or at least pretty small) function that results in an error.

First, does this fail if you simply create a function node and attempt to run it with static arguments? Or does it have to be in a workflow?

And can it be reduced, such as the following?

def create_header_dict_entry(in_file):
    img = nb.load(in_file)
    qap_dict = {"test": {"anatomical_header_info": {}}}
    return qap_dict

If this fails, it's much easier for us to try to reproduce this bug. If it doesn't, then finding the necessary lines to generate a failure should be instructive and help point us to the cause of the issue.

[sgiavasis]

Right- I got it to run with a minimal example similar to what you presented. I started re-introducing the code until I hit the error again.

Interestingly, it started occurring again with the inclusion of the iteration over the "info_labels" list. (The list definition where 'info_labels' is set goes through fine). I reduced the contents of that loop to only a plain, static print statement (to run in each iteration) and the error persisted. I then removed the loop entirely and tried to print the list itself, and the error still happened- it seems any reference to that list trips it up.

I'm going to continue swapping things around to see what trips the error, but hopefully this can be helpful for you in the meantime.

Edit: to be more precise, all of the code between the beginning of the function and the info_labels iteration goes through- re-introducing the iteration causes the error, and I have not re-introduced any of the code below it at any point.

[mgxd]

@sgiavasis can you try replacing your list with something like:

x = [str('hello'), str('there')]

and then try printing/iterating through it

[sgiavasis]

Just tried it- unfortunately, the error still occurs.