MATLAB crashed while using zarrread function to read a big (2317GB) array

[jm9176]

I can consistently reproduce the crash on Windows.

zarrinfo is working fine.

Repro steps:

>> zarrinfo('https://noaa-nwm-retro-v2-zarr-pds.s3.amazonaws.com/elevation/')

ans = 

  struct with fields:

                chunks: [2×1 double]
            compressor: [1×1 struct]
                 dtype: '<f4'
            fill_value: 'NaN'
               filters: []
                 order: 'C'
                 shape: [2×1 double]
           zarr_format: 2
             node_type: 'array'
    x_ARRAY_DIMENSIONS: {2×1 cell}
           coordinates: 'latitude longitude'
             long_name: 'Feature Elevation'
         standard_name: 'Elevation'
                 units: 'meters'

>> out = zarrread('https://noaa-nwm-retro-v2-zarr-pds.s3.amazonaws.com/elevation/');
% MATLAB crashes


Abnormal termination:
std::terminate()

Current Thread: '' id 15904

Register State (captured):
  RAX = 00007fff2f6eb2c8  RBX = 00007fff2f6eb2c8
  RCX = 000000ee5c3faca0  RDX = 00007fff2f694920
  RSP = 000000ee5c3fabe0  RBP = 000000ee5c3fb970
  RSI = 0000000000000000  RDI = 00007fff2f6eb2c8
 
   R8 = 0000000000000003   R9 = 000000ee5c3fab58
  R10 = 0000000000000014  R11 = 0000000000000000
  R12 = 000000ee5c3fb878  R13 = 0000000000000000
  R14 = 00007fff2f6db1f8  R15 = 000000ee5c3fb2b0
 
  RIP = 00007fff2f5d21da  EFL = 00000202
 
   CS = 0033   FS = 0053   GS = 002b

Stack Trace (captured):
[  0] 0x00007fff2f5cb573 C:\Program Files\MATLAB\R2025a\bin\win64\libmwfl.dll+00046451 foundation::core::diag::thread_context::unspecified_bool+00000051
[  1] 0x00007fff2f5c99a8 C:\Program Files\MATLAB\R2025a\bin\win64\libmwfl.dll+00039336 foundation::core::diag::stacktrace_base::capture+00000024
[  2] 0x00007fff2f5cde1f C:\Program Files\MATLAB\R2025a\bin\win64\libmwfl.dll+00056863 foundation::core::diag::symbols::getSymbolAddress+00009903
[  3] 0x00007fff2f5d0cc7 C:\Program Files\MATLAB\R2025a\bin\win64\libmwfl.dll+00068807 foundation::core::diag::is_terminate_message_enabled+00000519
[  4] 0x00007ffee80c0ad8   C:\Program Files\MATLAB\R2025a\bin\win64\mcr.dll+00854744 mnFullExitFcn+00036712
[  5] 0x00007ffee80c02fd   C:\Program Files\MATLAB\R2025a\bin\win64\mcr.dll+00852733 mnFullExitFcn+00034701
[  6] 0x00007ffee80bc655   C:\Program Files\MATLAB\R2025a\bin\win64\mcr.dll+00837205 mnFullExitFcn+00019173
[  7] 0x00007ffee80c22f9   C:\Program Files\MATLAB\R2025a\bin\win64\mcr.dll+00860921 mnSetSegvServiceUrl+00000185
[  8] 0x00007fff132e3b58 C:\Program Files\MATLAB\R2025a\bin\win64\libmwfoundation_crash_handling.dll+00146264 foundation::crash_handling::set_terminate+00000200
[  9] 0x00007fffcf6bee12                   C:\WINDOWS\System32\ucrtbase.dll+00519698 terminate+00000034
[ 10] 0x00007fffa8d61d8d C:\Program Files\MATLAB\R2025a\bin\win64\VCRUNTIME140.dll+00007565 is_exception_typeof+00002909
[ 11] 0x00007fffa8d62d90 C:\Program Files\MATLAB\R2025a\bin\win64\VCRUNTIME140.dll+00011664 is_exception_typeof+00007008
[ 12] 0x00007fffa8d6f351 C:\Program Files\MATLAB\R2025a\bin\win64\VCRUNTIME140.dll+00062289 _CxxFrameHandler3+00000113
[ 13] 0x00007fffd2155cef                      C:\WINDOWS\SYSTEM32\ntdll.dll+00679151 _chkstk+00000303
[ 14] 0x00007fffd20ce8c6                      C:\WINDOWS\SYSTEM32\ntdll.dll+00125126 RtlFindCharInUnicodeString+00002710
[ 15] 0x00007fffd2104995                      C:\WINDOWS\SYSTEM32\ntdll.dll+00346517 RtlRaiseException+00000405
[ 16] 0x00007fffcf2c00ac                 C:\WINDOWS\System32\KERNELBASE.dll+00393388 RaiseException+00000108
[ 17] 0x00007fffa8d65267 C:\Program Files\MATLAB\R2025a\bin\win64\VCRUNTIME140.dll+00021095 CxxThrowException+00000151
[ 18] 0x00007fff2f60dd4a C:\Program Files\MATLAB\R2025a\bin\win64\libmwfl.dll+00318794 foundation::core::mem::manager::instance+00004474
[ 19] 0x00007fff3c461ec1 C:\Program Files\MATLAB\R2025a\bin\win64\MSVCP140.dll+00073409 std::_MP_Rem+00000801
[ 20] 0x00007fffcf6ba5ec                   C:\WINDOWS\System32\ucrtbase.dll+00501228 callnewh+00000044
[ 21] 0x00007fff0317029f C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+23134879 PyInit__tensorstore+21296799
[ 22] 0x00007fff0307fcf0 C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+22150384 PyInit__tensorstore+20312304
[ 23] 0x00007fff01eb59da C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+03496410 PyInit__tensorstore+01658330
[ 24] 0x00007fff02795c25 C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+12803109 PyInit__tensorstore+10965029
[ 25] 0x00007fff0279518a C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+12800394 PyInit__tensorstore+10962314
[ 26] 0x00007fff02ed2894 C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+20392084 PyInit__tensorstore+18554004
[ 27] 0x00007fff02ed477e C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+20399998 PyInit__tensorstore+18561918
[ 28] 0x00007fff02ed3258 C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\tensorstore\_tensorstore.cp311-win_amd64.pyd+20394584 PyInit__tensorstore+18556504
[ 29] 0x00007fffcf669333                   C:\WINDOWS\System32\ucrtbase.dll+00168755 recalloc+00000163
[ 30] 0x00007fffd03f259d                   C:\WINDOWS\System32\KERNEL32.DLL+00075165 BaseThreadInitThunk+00000029
[ 31] 0x00007fffd210af78                      C:\WINDOWS\SYSTEM32\ntdll.dll+00372600 RtlUserThreadStart+00000040

[jm9176]

Thanks, @krisfed, for investigating this issue.

She noticed a hang on Mac if ZarrPy is used directly, but if we try the same steps on Windows, then there is a crash, but the error message (before crashing) in the terminal is somewhat similar to that on Mac.

From PythonModule directory in this repo

>>> import ZarrPy
>>> kv = ZarrPy.createKVStore(True, "elevation/", "noaa-nwm-retro-v2-zarr-pds")
>>> d = ZarrPy.readZarr(kv, [0,0], [227904,2729077], [1,1])
E0702 13:38:33.581197 1251370 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 13:38:33.581265 1251370 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 13:38:33.581273 1251370 AuthCredentialsProvider:6146] static: Profile credentials parser could not load or parse a credentials or config file.
E0702 13:38:33.581288 1251370 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 13:38:33.581296 1251370 AuthCredentialsProvider:6146] Failed to build config profile collection from file at (/Users/kfedoren/.aws/config) : Invalid file path.
W0702 13:38:34.594254 1251559 aws_api.cc:6148] id=0x12b607bf0: Stream completed with error code 2092(Timed out waiting for first byte of HTTP response, after sending the full request.)
W0702 13:38:35.984225 1251560 aws_api.cc:6148] id=0x12b607bf0: Stream completed with error code 2092(Timed out waiting for first byte of HTTP response, after sending the full request.)
W0702 13:38:35.984279 1251560 exp-backoff-strategy:1036] id=0x600001f2c230: token 0x11b72ebf0 has exhausted allowed retries. Retry count 1 max retries 1
E0702 13:38:35.984285 1251560 aws_api.cc:6148] id=0x12b607bf0: Stream completed, retries have been exhausted.
E0702 13:38:35.984314 1251560 aws_api.cc:6148] (id=0x12b607bf0) IMDS client failed to update the token from IMDS.
W0702 13:38:36.985639 1251562 aws_api.cc:6148] id=0x12b607bf0: Stream completed with error code 2092(Timed out waiting for first byte of HTTP response, after sending the full request.)
W0702 13:38:38.092120 1251562 aws_api.cc:6148] id=0x12b607bf0: Stream completed with error code 2092(Timed out waiting for first byte of HTTP response, after sending the full request.)
W0702 13:38:38.092189 1251562 exp-backoff-strategy:1036] id=0x600001f2c230: token 0x12b608e90 has exhausted allowed retries. Retry count 1 max retries 1
E0702 13:38:38.092198 1251562 aws_api.cc:6148] id=0x12b607bf0: Stream completed, retries have been exhausted.
E0702 13:38:38.092213 1251562 AuthCredentialsProvider:6146] (id=0x600002134400) Default chain credentials provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)

Additionally, the partial reading on the /elevation array works fine, but the above can be seen here as well

>>> kv = ZarrPy.createKVStore(True, "elevation/", "noaa-nwm-retro-v2-zarr-pds")
>>> ZarrPy.readZarr(kv, [0,0], [10,10], [1,1])
E0702 14:30:56.212430 1334294 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 14:30:56.212469 1334294 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/credentials' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 14:30:56.212474 1334294 AuthCredentialsProvider:6146] static: Profile credentials parser could not load or parse a credentials or config file.
E0702 14:30:56.212485 1334294 common-io:5] static: Failed to open file. path:'/Users/kfedoren/.aws/config' mode:'rb' errno:2 aws-error:44(AWS_ERROR_FILE_INVALID_PATH)
E0702 14:30:56.212489 1334294 AuthCredentialsProvider:6146] Failed to build config profile collection from file at (/Users/kfedoren/.aws/config) : Invalid file path.
W0702 14:30:57.214437 1334570 aws_api.cc:6148] id=0x14b7b3810: Stream completed with error code 2092(Timed out waiting for first byte of HTTP response, after sending the full request.)
W0702 14:30:57.298151 1334569 aws_api.cc:6148] id=0x14b7b3810: Stream completed with error code 1051(socket is closed.)
W0702 14:30:57.298203 1334569 exp-backoff-strategy:1036] id=0x600000878620: token 0x14b727d70 has exhausted allowed retries. Retry count 1 max retries 1
E0702 14:30:57.298211 1334569 aws_api.cc:6148] id=0x14b7b3810: Stream completed, retries have been exhausted.
E0702 14:30:57.298244 1334569 aws_api.cc:6148] (id=0x14b7b3810) IMDS client failed to update the token from IMDS.
W0702 14:30:57.299477 1334572 aws_api.cc:6148] id=0x14b7b3810: Stream completed with error code 1051(socket is closed.)
W0702 14:30:57.676999 1334569 aws_api.cc:6148] id=0x14b7b3810: Stream completed with error code 1051(socket is closed.)
W0702 14:30:57.677046 1334569 exp-backoff-strategy:1036] id=0x600000878620: token 0x12d0de4b0 has exhausted allowed retries. Retry count 1 max retries 1
E0702 14:30:57.677054 1334569 aws_api.cc:6148] id=0x14b7b3810: Stream completed, retries have been exhausted.
E0702 14:30:57.677065 1334569 AuthCredentialsProvider:6146] (id=0x600003665080) Default chain credentials provider failed to source credentials with error 6153(aws-c-auth: AWS_AUTH_CREDENTIALS_PROVIDER_IMDS_SOURCE_FAILURE, Valid credentials could not be sourced by the IMDS provider)
array([[ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08],
       [ 36.95, 191.6 , 184.31, 183.66, 183.66, 306.96, 275.07, 330.54,
        252.06, 304.08]], dtype=float32)
>>> 

Adding additional info related to the zarr library behavior:

Here, I get a proper error message due to the size of the array itself. However, the crash behavior that I get from MATLAB is incorrect and Kris is looking further to see what can be done about that.

Python 3.11.2 (tags/v3.11.2:878ead1, Feb  7 2023, 16:38:35) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import zarr
>>> z = zarr.open(r'https://noaa-nwm-retro-v2-zarr-pds.s3.amazonaws.com/elevation/',mode='r')
>>> print(z.shape)
(227904, 2729077)

>>> print(z.dtype)
float32

>>> print(z[1:10,1:10])
[[191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]
 [191.6  184.31 183.66 183.66 306.96 275.07 330.54 252.06 304.08]]

>>> print(z[:])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\zarr\core.py", line 799, in __getitem__
    result = self.get_basic_selection(pure_selection, fields=fields)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\zarr\core.py", line 925, in get_basic_selection
    return self._get_basic_selection_nd(selection=selection, out=out, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\zarr\core.py", line 967, in _get_basic_selection_nd
    return self._get_selection(indexer=indexer, out=out, fields=fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\ssohal\AppData\Local\Programs\Python\Python311\Lib\site-packages\zarr\core.py", line 1331, in _get_selection
    out = np.empty_like(
          ^^^^^^^^^^^^^^
numpy._core._exceptions._ArrayMemoryError: Unable to allocate 2.26 TiB for an array with shape (227904, 2729077) and data type float32
>>>