HDF5 file created with h5py can't be opened by h5py
Question:
I created an HDF5 file apparently without any problems, under Ubuntu 12.04 (32bit version), using Anaconda as Python distribution and writing in ipython notebooks. The underlying data are all numpy arrays. For example,
import numpy as np
import h5py
f = h5py.File('myfile.hdf5','w')
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
If I try to open this file from a new iypthon notebook, though, I get an error message:
f = h5py.File('myfile.hdf5', "r")
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-4-b64ac5089cd4> in <module>()
----> 1 f = h5py.File(file_name, "r")
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, driver, libver, userblock_size, **kwds)
220
221 fapl = make_fapl(driver, libver, **kwds)
--> 222 fid = make_fid(name, mode, userblock_size, fapl)
223
224 Group.__init__(self, fid)
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in make_fid(name, mode, userblock_size, fapl, fcpl)
77
78 if mode == 'r':
---> 79 fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl)
80 elif mode == 'r+':
81 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/h5f.so in h5py.h5f.open (h5py/h5f.c:1741)()
IOError: Unable to open file (Unable to find a valid file signature)
Can you tell me what that missing file signature is? Did I miss something when I created the file?
Answers:
Since we resolved the issue in the comments on my question, I’m writing the results out here to mark it as solved.
The main problem was that I forgot to close the file after I created it. There would have been two simple options, either:
import numpy as np
import h5py
f = h5py.File('myfile.hdf5','w')
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
f.close()
or, my favourite because the file is closed automatically:
import numpy as np
import h5py
with h5py.File('myfile.hdf5','w') as f:
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
I was working with https://github.com/matterport/Mask_RCNN and produced the same Error:
Traceback (most recent call last):
File "detection.py", line 42, in <module>
model.load_weights(model_path, by_name=True)
File "/home/michael/Bachelor/important/Cable-detection/Mask_RCNN-2.1/samples/cable/mrcnn/model.py", line 2131, in load_weights
f = h5py.File(filepath, mode='r')
File "/home/michael/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 271, in __init__
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/home/michael/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-s_7obrrg-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-s_7obrrg-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-s_7obrrg-build/h5py/h5f.c:2117)
OSError: Unable to open file (Addr overflow, addr = 800, size=8336, eoa=2144)
HDF5: infinite loop closing library
D,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
The solution above worked for me too. I interrupted the training at a random point, therefor the .hdf5
file was not closed properly and could not be opened afterward.
File "S2_network_training.py", line 31, in <module>
f = h5py.File(dataset_path,'r')
File "/home/vislab-002/anaconda3/envs/XYZNet/lib/python3.7/site-packages/h5py/_hl/files.py", line 427, in __init__
swmr=swmr)
File "/home/vislab-002/anaconda3/envs/XYZNet/lib/python3.7/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (bad object header version number)
I checked to make sure i closed the file but i still get the error. Help please
I created an HDF5 file apparently without any problems, under Ubuntu 12.04 (32bit version), using Anaconda as Python distribution and writing in ipython notebooks. The underlying data are all numpy arrays. For example,
import numpy as np
import h5py
f = h5py.File('myfile.hdf5','w')
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
If I try to open this file from a new iypthon notebook, though, I get an error message:
f = h5py.File('myfile.hdf5', "r")
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-4-b64ac5089cd4> in <module>()
----> 1 f = h5py.File(file_name, "r")
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in __init__(self, name, mode, driver, libver, userblock_size, **kwds)
220
221 fapl = make_fapl(driver, libver, **kwds)
--> 222 fid = make_fid(name, mode, userblock_size, fapl)
223
224 Group.__init__(self, fid)
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/_hl/files.pyc in make_fid(name, mode, userblock_size, fapl, fcpl)
77
78 if mode == 'r':
---> 79 fid = h5f.open(name, h5f.ACC_RDONLY, fapl=fapl)
80 elif mode == 'r+':
81 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
/home/sarah/anaconda/lib/python2.7/site-packages/h5py/h5f.so in h5py.h5f.open (h5py/h5f.c:1741)()
IOError: Unable to open file (Unable to find a valid file signature)
Can you tell me what that missing file signature is? Did I miss something when I created the file?
Since we resolved the issue in the comments on my question, I’m writing the results out here to mark it as solved.
The main problem was that I forgot to close the file after I created it. There would have been two simple options, either:
import numpy as np
import h5py
f = h5py.File('myfile.hdf5','w')
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
f.close()
or, my favourite because the file is closed automatically:
import numpy as np
import h5py
with h5py.File('myfile.hdf5','w') as f:
group = f.create_group('a_group')
group.create_dataset(name='matrix', data=np.zeros((10, 10)), chunks=True, compression='gzip')
I was working with https://github.com/matterport/Mask_RCNN and produced the same Error:
Traceback (most recent call last):
File "detection.py", line 42, in <module>
model.load_weights(model_path, by_name=True)
File "/home/michael/Bachelor/important/Cable-detection/Mask_RCNN-2.1/samples/cable/mrcnn/model.py", line 2131, in load_weights
f = h5py.File(filepath, mode='r')
File "/home/michael/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 271, in __init__
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/home/michael/.local/lib/python3.6/site-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-s_7obrrg-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-s_7obrrg-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-s_7obrrg-build/h5py/h5f.c:2117)
OSError: Unable to open file (Addr overflow, addr = 800, size=8336, eoa=2144)
HDF5: infinite loop closing library
D,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
The solution above worked for me too. I interrupted the training at a random point, therefor the .hdf5
file was not closed properly and could not be opened afterward.
File "S2_network_training.py", line 31, in <module>
f = h5py.File(dataset_path,'r')
File "/home/vislab-002/anaconda3/envs/XYZNet/lib/python3.7/site-packages/h5py/_hl/files.py", line 427, in __init__
swmr=swmr)
File "/home/vislab-002/anaconda3/envs/XYZNet/lib/python3.7/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (bad object header version number)
I checked to make sure i closed the file but i still get the error. Help please