Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: #255

Open
rcpeene opened this issue Jan 6, 2025 · 3 comments · May be fixed by #256
Open

[Bug]: #255

rcpeene opened this issue Jan 6, 2025 · 3 comments · May be fixed by #256
Assignees
@rly @mavaylon1
Labels
category: bug errors in the code or code behavior priority: high impacts proper operation or use of feature important to most users
Milestone
0.11.0

Comments

@rcpeene
Copy link

rcpeene commented Jan 6, 2025

What happened?

I am trying to open a zarr nwb in order to append CCF information to the electrodes table, but it fails to open.

Note that .zmetadata is present in the Zarr NWB file. It's worth mentioning that I generated this hdmf-zarr==0.8.0 but and now trying to open it with latest (0.10.0), perhaps that is causing this.

Steps to Reproduce

Run the following code snippet, where there is a Zarr NWB present at `scratch_nwb_path`


    print('Reading NWB in append mode:', scratch_nwb_path)
    print(os.listdir(scratch_folder))
    if scratch_nwb_path.is_dir():
        print(os.listdir(scratch_nwb_path))
    with NWBZarrIO(str(scratch_nwb_path), mode='a') as read_io:
        nwb = read_io.read()

Traceback

Reading NWB in append mode: ../scratch/ecephys_747816_2024-08-14_13-57-37_experiment1_recording1.nwb
['ecephys_747816_2024-08-14_13-57-37_experiment1_recording1.nwb']
['file_create_date', 'intervals', 'analysis', 'specifications', 'stimulus', '.zattrs', 'general', 'acquisition', '.zmetadata', '.zgroup', 'processing', 'timestamps_reference_time', 'identifier', 'units', 'session_description', 'session_start_time']
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 566, in __open_file_consolidated
    return zarr.open_consolidated(
  File "/opt/conda/lib/python3.9/site-packages/zarr/convenience.py", line 1360, in open_consolidated
    meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 3046, in __init__
    meta = json_loads(self.store[metadata_key])
  File "/opt/conda/lib/python3.9/site-packages/zarr/storage.py", line 1120, in __getitem__
    raise KeyError(key)
KeyError: '.zmetadata'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/capsule/code/run_capsule.py", line 140, in <module>
    if __name__ == "__main__": run()
  File "/root/capsule/code/run_capsule.py", line 124, in run
    nwb = read_io.read()
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/io.py", line 56, in read
    f_builder = self.read_builder()
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1470, in read_builder
    f_builder = self.__read_group(self.__file, ROOT_NAME)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1548, in __read_group
    sub_builder = self.__read_group(sub_group, sub_name)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1548, in __read_group
    sub_builder = self.__read_group(sub_group, sub_name)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1557, in __read_group
    self.__read_links(zarr_obj=zarr_obj, parent=ret)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 1576, in __read_links
    target_name, target_zarr_obj = self.resolve_ref(link)
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 822, in resolve_ref
    target_zarr_obj = self.__open_file_consolidated(
  File "/opt/conda/lib/python3.9/site-packages/hdmf_zarr/backend.py", line 573, in __open_file_consolidated
    return zarr.open(
  File "/opt/conda/lib/python3.9/site-packages/zarr/convenience.py", line 137, in open
    raise PathNotFoundError(path)
zarr.errors.PathNotFoundError: nothing found at path ''
(base) root@a09d29db9803:~/capsule/code# 


### Operating System

Windows

### Python Version

3.9

### Package Versions

_No response_
@alejoe91
Copy link
Collaborator

alejoe91 commented Jan 7, 2025

Getting the same error. Actually, in my case this also happens with a round-trip test, where an NWB-zarr file is generated and then read again with hdmf-zarr 0.10.0

See PR here: SpikeInterface/spikeinterface#3598

@rly rly assigned rly and mavaylon1 Jan 7, 2025
@rly
Copy link
Contributor

rly commented Jan 7, 2025

@rcpeene @alejoe91 Thank you for the report. We will take a look and also make sure we include an appropriate round-trip test to catch errors like this in the future.

@rly rly added category: bug errors in the code or code behavior priority: high impacts proper operation or use of feature important to most users labels Jan 7, 2025
@rly rly added this to the 0.11.0 milestone Jan 7, 2025
@rly
Copy link
Contributor

rly commented Jan 7, 2025

The error appears to come from reading a reference from a zarr store that is not the current working directory. Below is a minimal working example. It fails when reading from "test/test.nwb" but succeeds when reading from "test.nwb". I will continue to investigate.

from pynwb.testing.mock.file import mock_NWBFile
from pynwb.testing.mock.device import mock_Device
from pynwb.testing.mock.ecephys import mock_ElectricalSeries, mock_ElectrodeGroup
import numpy as np

from hdmf_zarr import NWBZarrIO

nwbfile = mock_NWBFile()
device = mock_Device(name="probe")
nwbfile.add_device(device)
nwbfile.add_electrode_column(name="channel_name", description="channel name")

electrode_group = mock_ElectrodeGroup(device=device)
nwbfile.add_electrode_group(electrode_group)
nwbfile.add_electrode(group=electrode_group, location="unknown", channel_name="channel_name1")

electrode_region = nwbfile.create_electrode_table_region(region=[0], description="electrodes")
electrical_series = mock_ElectricalSeries(
    name="electrical_series_name",
    electrodes=electrode_region,
    data=np.array([1, 2]),
    rate=1000.0,
)
nwbfile.add_acquisition(electrical_series)

import os
os.makedirs("test", exist_ok=True)
with NWBZarrIO("test/test.nwb", mode="w") as io:
    io.write(nwbfile)

io = NWBZarrIO("test/test.nwb", mode="r")
read_nwbfile = io.read()
io.close()

Error:

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/convenience.py:1362, in open_consolidated(store, metadata_key, mode, **kwargs)
   1361 # setup metadata store
-> 1362 meta_store = ConsolidatedStoreClass(store, metadata_key=metadata_key)
   1364 # pass through

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/storage.py:3045, in ConsolidatedMetadataStore.__init__(self, store, metadata_key)
   3044 # retrieve consolidated metadata
-> 3045 meta = json_loads(self.store[metadata_key])
   3047 # check format of consolidated metadata

File ~/miniconda3/envs/temp/lib/python3.12/site-packages/zarr/storage.py:1120, in DirectoryStore.__getitem__(self, key)
   1119 else:
-> 1120     raise KeyError(key)
...
    137     return open_group(_store, mode=mode, **kwargs)
    138 else:
--> 139     raise PathNotFoundError(path)

PathNotFoundError: nothing found at path ''

@rly rly linked a pull request Jan 8, 2025 that will close this issue
Draft
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rly rly

@mavaylon1 mavaylon1

Labels
category: bug errors in the code or code behavior priority: high impacts proper operation or use of feature important to most users
Projects
None yet
Milestone
0.11.0
Development

Successfully merging a pull request may close this issue.

[WIP] Fix relative references when store not in cwd hdmf-dev/hdmf-zarr
4 participants
@rly @alejoe91 @mavaylon1 @rcpeene