[Bug]: File written with MatNWB fails to open with PyNWB

[arnodelorme]

What happened?

This file saved using the NwbFile object

https://nemar-dev.ucsd.edu/dataexplorer/download?filepath=/expanse/projects/nemar/openneuro/nwb_example_files/sub-005_ieeg.nwb

Can be reimported in MATLAB using the same object. However, it fails to be reimported in Python. @dungscout96, you used the two different Python libraries right? Can you explain further?

Steps to Reproduce

Try importing the file in Python.

Error Message

Operating System

macOS

Matlab Version

2024b

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[ehennestad]

Hi,

I have tried to read the file with both MatNWB and PyNWB (v2.8.1), and both worked. What version of PyNWB do you use?
(Use import pynwb; pynwb.__version__ to check)

[arnodelorme]

@dungscout96 can you comment?

[arnodelorme]

Oh this is actually this file https://testspeedeegdash.s3.us-east-2.amazonaws.com/sub-002_task-FaceRecognition_eeg.nwb -- I have just imported it in MATLAB and it was fine (it was exported using MATLAB tools). However, it fails in Python.

[dungscout96]

Here's the error I'm seeing using pynwb version 2.8.3:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1369, in ObjectMapper.construct(self, **kwargs)
   1368 try:
-> 1369     obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),
   1370                                  **kwargs)
   1371 except Exception as ex:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1382, in ObjectMapper.__new_container__(self, cls, container_source, parent, object_id, **kwargs)
   1380 # obj has been created and is in construction mode, indicating that the object is being constructed by
   1381 # the automatic construct process during read, rather than by the user
-> 1382 obj.__init__(**kwargs)
   1383 obj._in_construct_mode = False  # reset to False to indicate that the construction of the object is complete

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:667, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
--> 667     pargs = _check_args(args, kwargs)
    668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:660, in docval.<locals>.dec.<locals>._check_args(args, kwargs)
    659         msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
--> 660         raise ExceptionType(msg)
    662 return parsed['args']

TypeError: ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')

The above exception was the direct cause of the following exception:

ConstructError                            Traceback (most recent call last)
Cell In[1], line 3
      1 from pynwb import NWBHDF5IO
      2 with NWBHDF5IO('/Users/dtyoung/Downloads/sub-002_task-FaceRecognition_eeg.nwb', 'r') as io:
----> 3     nwbfile = io.read()

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/pynwb/__init__.py:458, in NWBHDF5IO.read(self, **kwargs)
    455         raise TypeError("NWB version %s not supported. PyNWB supports NWB files version 2 and above." %
    456                         str(file_version_str))
    457 # read the file
--> 458 file = super().read(**kwargs)
    459 return file

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/backends/hdf5/h5tools.py:501, in HDF5IO.read(self, **kwargs)
    498     raise UnsupportedOperation("Cannot read from file %s in mode '%s'. Please use mode 'r', 'r+', or 'a'."
    499                                % (self.source, self.__mode))
    500 try:
--> 501     return super().read(**kwargs)
    502 except UnsupportedOperation as e:
    503     if str(e) == 'Cannot build data. There are no values.':  # pragma: no cover

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/backends/io.py:60, in HDMFIO.read(self, **kwargs)
     57 if all(len(v) == 0 for v in f_builder.values()):
     58     # TODO also check that the keys are appropriate. print a better error message
     59     raise UnsupportedOperation('Cannot build data. There are no values.')
---> 60 container = self.__manager.construct(f_builder)
     61 container.read_io = self
     62 if self.herd_path is not None:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:286, in BuildManager.construct(self, **kwargs)
    282     result = self.__type_map.construct(builder, self, parent)
    283 else:
    284     # we are at the top of the hierarchy,
    285     # so it must be time to resolve parents
--> 286     result = self.__type_map.construct(builder, self, None)
    287     self.__resolve_parents(result)
    288 self.prebuilt(result, builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:827, in TypeMap.construct(self, **kwargs)
    825     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    826 else:
--> 827     return obj_mapper.construct(builder, build_manager, parent)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1339, in ObjectMapper.construct(self, **kwargs)
   1337 cls = manager.get_cls(builder)
   1338 # gather all subspecs
-> 1339 subspecs = self.__get_subspec_values(builder, self.spec, manager)
   1340 # get the constructor argument that each specification corresponds to
   1341 const_args = dict()

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1319, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1316     continue
   1317 if dt is None:
   1318     # recurse
-> 1319     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1320 else:
   1321     ret[subspec] = manager.construct(sub_builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1319, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1316     continue
   1317 if dt is None:
   1318     # recurse
-> 1319     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1320 else:
   1321     ret[subspec] = manager.construct(sub_builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1311, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1309     sub_builder = builder_dt.get(dt)
   1310     if sub_builder is not None:
-> 1311         sub_builder = self.__flatten(sub_builder, subspec, manager)
   1312         ret[subspec] = sub_builder
   1313 else:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1324, in ObjectMapper.__flatten(self, sub_builder, subspec, manager)
   1323 def __flatten(self, sub_builder, subspec, manager):
-> 1324     tmp = [manager.construct(b) for b in sub_builder]
   1325     if len(tmp) == 1 and not subspec.is_many():
   1326         tmp = tmp[0]

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1324, in <listcomp>(.0)
   1323 def __flatten(self, sub_builder, subspec, manager):
-> 1324     tmp = [manager.construct(b) for b in sub_builder]
   1325     if len(tmp) == 1 and not subspec.is_many():
   1326         tmp = tmp[0]

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:282, in BuildManager.construct(self, **kwargs)
    280 if parent_builder is not None:
    281     parent = self._get_proxy_builder(parent_builder)
--> 282     result = self.__type_map.construct(builder, self, parent)
    283 else:
    284     # we are at the top of the hierarchy,
    285     # so it must be time to resolve parents
    286     result = self.__type_map.construct(builder, self, None)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:827, in TypeMap.construct(self, **kwargs)
    825     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    826 else:
--> 827     return obj_mapper.construct(builder, build_manager, parent)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1373, in ObjectMapper.construct(self, **kwargs)
   1371 except Exception as ex:
   1372     msg = 'Could not construct %s object due to: %s' % (cls.__name__, ex)
-> 1373     raise ConstructError(builder, msg) from ex
   1374 return obj

ConstructError: (root/general/extracellular_ephys/all_electrodes GroupBuilder {'attributes': {'description': 'all electrodes', 'location': array([], dtype=object), 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': '2a87fe4b-0cc3-458d-988b-34b55a2826d9'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/all_electrodes/device LinkBuilder {'builder': root/general/devices/array GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '5f38dc95-4ad3-4700-a70a-58f803b99c82'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, "Could not construct ElectrodeGroup object due to: ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')")

Here's the code I'm running:

from pynwb import NWBHDF5IO
with NWBHDF5IO('/Users/dtyoung/Downloads/sub-002_task-FaceRecognition_eeg.nwb', 'r') as io:
    nwbfile = io.read()

[ehennestad]

Thanks for providing the file and the error trace. I can reproduce this now!

@stephprince , Can you weigh in:

There is one ElectrodeGroup in the h5 location "root/general/extracellular_ephys/all_electrodes".

The location property of the ElectrodeGroup is a string in MatNWB, but it is empty which should be fine. The corresponding dataset attribute in the HDF5 file also appears to be a string type (see screenshot below), but the PyNWB error says that it gets an ndarray value:

ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')")

[oruebel]

The corresponding dataset attribute in the HDF5 file also appears to be a string type (see screenshot below), but the PyNWB error says that it gets an ndarray value:

Location should be a Scalar attribute of type string. However, according to the screenshot you shared it appears to be an array of strings with 0 elements instead. If the string is empty then it should be e an empty scalar string not an empty array.

[ehennestad]

Sorry for the delay!

This appears to be a bug related to exporting files from MatNWB.

@arnodelorme @dungscout96 Is it an option to re-export the file? I have added a fix in PR #652 and I can notify you when this is merged into main.

Alternatively, you can also immediately re-export the file and add a non-empty character vector to the location property of types.core.ElectrodeGroup

If re-exporting is not an option, you could create a small python function to modify the file in place before reading it with pynwb:

import h5py

nwbFile = "path/to/nwbfile"

with h5py.File(nwbFile, "r+") as f:
    # Get the dataset with the electrode group
    dset = f["/general/extracellular_ephys/all_electrodes"]

    # Delete the old attribute for location
    del dset.attrs["location"]

    # Create a *scalar* string attribute with the empty string as its value.
    # h5py supports variable-length strings via h5py.special_dtype(vlen=str), but for a scalar
    # you can simply pass a Python string (which is implicitly treated as a scalar string).
    dset.attrs.create("location", "", dtype=h5py.special_dtype(vlen=str))

After doing this, there appears to be another problem that blocks reading the file with PyNWB. The NWB file does not have the required property session_start_time. Enforcing required properties was recently added in the v2.7.0 release of MatNWB, but I see that this file was created with an older release/version.

Please let me know if any of these suggestions are helpful or if you need further assistance.