Filtering PyTables PerformanceWarning with warnings.filterwarnings() fails Question: There are a number of answers on this website detailing how one can ignore specific warnings in python (either by category or by providing a regex to match a warning message). However, none of these seem to work when I try try to suppress PerformanceWarnings coming from PyTables. …
How to save a list in a pandas dataframe cell to a HDF5 table format? Question: I have a dataframe that I want to save in the appendable format to a hdf5 file. The dataframe looks like this: column1 0 [0, 1, 2, 3, 4] And the code that replicates the issue is: import pandas …
Missing optional dependency 'tables'. In pandas to_hdf Question: following code is giving me error. import pandas as pd df = pd.DataFrame({‘a’ : [1,2,3]}) df.to_hdf(‘temp.h5′, key=’df’, mode=’w’) This is giving me error. Missing optional dependency ‘tables’. Use pip or conda to install tables. I already tried ImportError HDFStore requires PyTables No module named tables. Still the …
install HDF5 and pytables in ubuntu Question: I am trying to install tables package in Ubuntu 14.04 but sems like it is complaining. I am trying to install it using PyCharm and its package installer, however seems like it is complaining about HDF5 package. However, seems like I cannnot find any hdf5 package to install …
Is there an analysis speed or memory usage advantage to using HDF5 for large array storage (instead of flat binary files)? Question: I am processing large 3D arrays, which I often need to slice in various ways to do a variety of data analysis. A typical “cube” can be ~100GB (and will likely get larger …
Convert large csv to hdf5 Question: I have a 100M line csv file (actually many separate csv files) totaling 84GB. I need to convert it to a HDF5 file with a single float dataset. I used h5py in testing without any problems, but now I can’t do the final dataset without running out of memory. …
Improve pandas (PyTables?) HDF5 table write performance Question: I’ve been using pandas for research now for about two months to great effect. With large numbers of medium-sized trace event datasets, pandas + PyTables (the HDF5 interface) does a tremendous job of allowing me to process heterogenous data using all the Python tools I know and …
HDF5 taking more space than CSV? Question: Consider the following example: Prepare the data: import string import random import pandas as pd matrix = np.random.random((100, 3000)) my_cols = [random.choice(string.ascii_uppercase) for x in range(matrix.shape[1])] mydf = pd.DataFrame(matrix, columns=my_cols) mydf[‘something’] = ‘hello_world’ Set the highest compression possible for HDF5: store = pd.HDFStore(‘myfile.h5′,complevel=9, complib=’bzip2’) store[‘mydf’] = mydf store.close() …
