Streamline failures for big-endian

[matthew-brett]

Sorry to be slow to notice, but the new streamline code is failing on our big-endian PPC buildbot machine:

http://nipy.bic.berkeley.edu/builders/nibabel-py2.7-debian-ppc/builds/84/steps/shell_5/logs/stdio

For example:

======================================================================
ERROR: test_save_complex_file (nibabel.streamlines.tests.test_streamlines.TestLoadSave)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-debian-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/tests/test_streamlines.py", line 260, in test_save_complex_file
    tfile = nib.streamlines.load(filename, lazy_load=False)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-debian-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/__init__.py", line 91, in load
    return tractogram_file.load(fileobj, lazy_load=lazy_load)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-debian-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/trk.py", line 390, in load
    arr_seqs = create_arraysequences_from_generator(trk_reader, n=3)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-debian-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/array_sequence.py", line 375, in create_arraysequences_from_generator
    for data in gen:
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-debian-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/trk.py", line 691, in _read
    buffer=f.read(nb_pts * pts_and_scalars_size))
TypeError: buffer is too small for requested array

@MarcCote - any ideas? Would you like a login to that machine to debug?

[matthew-brett]

I got further by making this change:

diff --git a/nibabel/streamlines/trk.py b/nibabel/streamlines/trk.py
index 9eeef0f..245f05c 100644
--- a/nibabel/streamlines/trk.py
+++ b/nibabel/streamlines/trk.py
@@ -227,7 +227,7 @@ def decode_value_from_name(encoded_name):

 def create_empty_header():
     """ Return an empty compliant TRK header. """
-    header = np.zeros(1, dtype=header_2_dtype)
+    header = np.zeros(1, dtype=header_2_dtype).newbyteorder('<')

     # Default values
     header[Field.MAGIC_NUMBER] = TrkFile.MAGIC_NUMBER

That has the effect of making sure that the header is written as little-endian, on big-endian systems, which I think is your intention.

But - after I've done this, I get errors on the round-trip tests, like this:

======================================================================
FAIL: test_save_simple_file (nibabel.streamlines.tests.test_streamlines.TestLoadSave)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/tests/te
    assert_tractogram_equal(tfile.tractogram, tractogram)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/tests/te
    t2.data_per_streamline, t2.data_per_point)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/streamlines/tests/te
    assert_arrays_equal(tractogram.streamlines, streamlines)
  File "/home/buildslave/debian-ppc-64/nibabel-py2_7-ppc/build/venv/local/lib/python2.7/site-packages/nibabel/testing/__init__.py"
    assert_array_equal(arr1, arr2)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 718, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 644, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal

(mismatch 100.0%)
 x: array([[-0.5, -0.5, -0.5]], dtype=float32)
 y: array([[ 0.,  1.,  2.]], dtype=float32)

Copying the written file across to a little-endian system confirms that the points have been written incorrectly as all 0.5 coordinates. It's getting late here, but, do you have any ideas about this one?

[matthew-brett]

OK - I think I've tracked these down - see #473