Accessing all parameters of a GRIB2 file using pygrib
Question:
I am trying to read a GRIB2 file using pygrib. I manage to open files and read data without problem, but I need to automate the process, and unfortunately each file contains multiple very similar fields:
1:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 214:fcst time 0-1 hrs (avg):from 201705200000
2:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 224:fcst time 0-1 hrs (avg):from 201705200000
3:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 234:fcst time 0-1 hrs (avg):from 201705200000
4:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 10:fcst time 0-1 hrs (avg):from 201705200000
5:Total Cloud Cover:% (instant):regular_ll:unknown:level 0 244:fcst time 1 hrs:from 201705200000
6:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 211:fcst time 0-1 hrs (avg):from 201705200000
The only difference (as seen by pygrib) for these file is the field typeOfFirstFixedSurface, but I have no idea what this field (and cannot find the relevant information on the site where I got the grib2 files). I have looked at all the key/value parameters for every messages, and did not find any other useful information I could use to differentiate the fields….
However, when using Panoply, I see much more parameters, e.g.:
float Total_cloud_cover_convective_cloud(time=1, lat=721, lon=1440);
:long_name = "Total cloud cover @ Convective cloud layer";
:units = "%";
:abbreviation = "TCDC";
:missing_value = NaNf; // float
:grid_mapping = "LatLon_Projection";
:coordinates = "reftime time lat lon ";
:Grib_Variable_Id = "VAR_0-6-1_L244";
:Grib2_Parameter = 0, 6, 1; // int
:Grib2_Parameter_Discipline = "Meteorological products";
:Grib2_Parameter_Category = "Cloud";
:Grib2_Parameter_Name = "Total cloud cover";
:Grib2_Level_Type = "Convective cloud layer";
:Grib2_Generating_Process_Type = "Forecast";
I could definitively use the long_name or Grib_Variable_Id fields to differentiate between messages, but I cannot access these “parameters” using pygrib.
Is there a way to access these parameters using pygrib?
Answers:
I had lots of problems with figuring out this particular format and tools which could help me with reading/parsing it… Eventually, I ended up with using IRIS which ended up in extremely messy setup. I did this a year and a half ago on Ubuntu machine with Python 2.7. I am pasting the steps I needed to do in order to install IRIS and all possible dependencies, hopefully this will be of some use to you although there are probably lots of outdated versions in the whole setup…
Good luck!
# http://scitools.org.uk/iris/docs/latest/installing.html
# necessary steps for a clean machine
#
# sudo apt-get install gcc python-dev build-essential python-setuptools libpq-dev git unzip cmake
# pip install virtualenv
# pip install virtualenvwrapper
# mkdir ~/.virtualenvs
# nano ~/.bashrc
# export WORKON_HOME=$HOME/.virtualenvs
# source /usr/local/bin/virtualenvwrapper.sh
# . ~/.bashrc
# mkvirtualenv iris
pip install numpy
pip install biggus
sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran
# OR
# sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose
pip install scipy
# cartopy dependecies
pip install cython
# https://github.com/OSGeo/proj.4
wget http://download.osgeo.org/proj/proj-4.9.1.tar.gz
tar -xzf proj-4.9.1.tar.gz
cd proj-4.9.1/
./configure
make
make install
# OR
# sudo apt-get install libproj-dev
# sudo apt-get install libgeos-dev
# http://sourceforge.net/projects/pyke/files/pyke/
wget http://sourceforge.net/projects/pyke/files/pyke/1.1.1/pyke-1.1.1.zip/download
unzip download
cd pyke-1.1.1/
python setup.py build
python setup.py install
pip install cartopy
# netcdf4 dependecies
# https://code.google.com/p/netcdf4-python/wiki/UbuntuInstall
# wget https://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.16.tar (??????????????)
sudo apt-get install libhdf5-dev
pip install h5py
sudo apt-get install python-netcdf libnetcdf-dev libnetcdf4
git clone https://github.com/Unidata/netcdf4-python.git
cd netcdf4-python
python setup.py build
python setup.py install
# other iris dependencies
pip install cf_units
sudo apt-get install libudunits2-dev
nano ~/.bashrc
# check where exactly xml file is
export UDUNITS2_XML_PATH=/usr/local/share/doc/udunits/udunits2.xml
. ~/.bashrc
pip install pillow
# gribapi
# https://software.ecmwf.int/wiki/display/GRIB/GRIB+API+CMake+installation
wget https://software.ecmwf.int/wiki/download/attachments/3473437/grib_api-1.14.4-Source.tar.gz?api=v2
tar -xzf grib_api-1.14.4-Source.tar.gz?api=v2
mkdir build; cd build
cmake ../grib_api-1.14.0-Source -DENABLE_PYTHON=ON
make -j4
ctest -j4
make install
# OR
# sudo apt-get install libgrib-api-dev
# sudo apt-get install openjpeg-tools
cp -R /usr/local/lib/python2.7/site-packages/grib_api ~/.virtualenvs/iristest/lib/python2.7/site-packages/
# aaaand, here we go, iris!
git clone https://github.com/SciTools/iris.git
cd iris
python setup.py build
python setup.py install
# rejoice!
# pip freeze output:
# Biggus==0.12.0
# Cartopy==0.13.0
# cf-units==1.0.0
# Cython==0.23.4
# h5py==2.5.0
# Iris==1.10.0.dev0
# netCDF4==1.2.2
# numpy==1.10.2
# Pillow==3.0.0
# pyke==1.1.1
# pyshp==1.2.3
# scipy==0.16.1
# Shapely==1.5.13
# six==1.10.0
In order to install iris, using the Anaconda distribution, I did:
conda install -c conda-forge iris
and all required packages were installed at once without problems.
I have Anaconda 3 version.
I am trying to read a GRIB2 file using pygrib. I manage to open files and read data without problem, but I need to automate the process, and unfortunately each file contains multiple very similar fields:
1:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 214:fcst time 0-1 hrs (avg):from 201705200000
2:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 224:fcst time 0-1 hrs (avg):from 201705200000
3:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 234:fcst time 0-1 hrs (avg):from 201705200000
4:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 10:fcst time 0-1 hrs (avg):from 201705200000
5:Total Cloud Cover:% (instant):regular_ll:unknown:level 0 244:fcst time 1 hrs:from 201705200000
6:Total Cloud Cover:% (avg):regular_ll:unknown:level 0 211:fcst time 0-1 hrs (avg):from 201705200000
The only difference (as seen by pygrib) for these file is the field typeOfFirstFixedSurface, but I have no idea what this field (and cannot find the relevant information on the site where I got the grib2 files). I have looked at all the key/value parameters for every messages, and did not find any other useful information I could use to differentiate the fields….
However, when using Panoply, I see much more parameters, e.g.:
float Total_cloud_cover_convective_cloud(time=1, lat=721, lon=1440);
:long_name = "Total cloud cover @ Convective cloud layer";
:units = "%";
:abbreviation = "TCDC";
:missing_value = NaNf; // float
:grid_mapping = "LatLon_Projection";
:coordinates = "reftime time lat lon ";
:Grib_Variable_Id = "VAR_0-6-1_L244";
:Grib2_Parameter = 0, 6, 1; // int
:Grib2_Parameter_Discipline = "Meteorological products";
:Grib2_Parameter_Category = "Cloud";
:Grib2_Parameter_Name = "Total cloud cover";
:Grib2_Level_Type = "Convective cloud layer";
:Grib2_Generating_Process_Type = "Forecast";
I could definitively use the long_name or Grib_Variable_Id fields to differentiate between messages, but I cannot access these “parameters” using pygrib.
Is there a way to access these parameters using pygrib?
I had lots of problems with figuring out this particular format and tools which could help me with reading/parsing it… Eventually, I ended up with using IRIS which ended up in extremely messy setup. I did this a year and a half ago on Ubuntu machine with Python 2.7. I am pasting the steps I needed to do in order to install IRIS and all possible dependencies, hopefully this will be of some use to you although there are probably lots of outdated versions in the whole setup…
Good luck!
# http://scitools.org.uk/iris/docs/latest/installing.html
# necessary steps for a clean machine
#
# sudo apt-get install gcc python-dev build-essential python-setuptools libpq-dev git unzip cmake
# pip install virtualenv
# pip install virtualenvwrapper
# mkdir ~/.virtualenvs
# nano ~/.bashrc
# export WORKON_HOME=$HOME/.virtualenvs
# source /usr/local/bin/virtualenvwrapper.sh
# . ~/.bashrc
# mkvirtualenv iris
pip install numpy
pip install biggus
sudo apt-get install libblas-dev liblapack-dev libatlas-base-dev gfortran
# OR
# sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-notebook python-pandas python-sympy python-nose
pip install scipy
# cartopy dependecies
pip install cython
# https://github.com/OSGeo/proj.4
wget http://download.osgeo.org/proj/proj-4.9.1.tar.gz
tar -xzf proj-4.9.1.tar.gz
cd proj-4.9.1/
./configure
make
make install
# OR
# sudo apt-get install libproj-dev
# sudo apt-get install libgeos-dev
# http://sourceforge.net/projects/pyke/files/pyke/
wget http://sourceforge.net/projects/pyke/files/pyke/1.1.1/pyke-1.1.1.zip/download
unzip download
cd pyke-1.1.1/
python setup.py build
python setup.py install
pip install cartopy
# netcdf4 dependecies
# https://code.google.com/p/netcdf4-python/wiki/UbuntuInstall
# wget https://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.16.tar (??????????????)
sudo apt-get install libhdf5-dev
pip install h5py
sudo apt-get install python-netcdf libnetcdf-dev libnetcdf4
git clone https://github.com/Unidata/netcdf4-python.git
cd netcdf4-python
python setup.py build
python setup.py install
# other iris dependencies
pip install cf_units
sudo apt-get install libudunits2-dev
nano ~/.bashrc
# check where exactly xml file is
export UDUNITS2_XML_PATH=/usr/local/share/doc/udunits/udunits2.xml
. ~/.bashrc
pip install pillow
# gribapi
# https://software.ecmwf.int/wiki/display/GRIB/GRIB+API+CMake+installation
wget https://software.ecmwf.int/wiki/download/attachments/3473437/grib_api-1.14.4-Source.tar.gz?api=v2
tar -xzf grib_api-1.14.4-Source.tar.gz?api=v2
mkdir build; cd build
cmake ../grib_api-1.14.0-Source -DENABLE_PYTHON=ON
make -j4
ctest -j4
make install
# OR
# sudo apt-get install libgrib-api-dev
# sudo apt-get install openjpeg-tools
cp -R /usr/local/lib/python2.7/site-packages/grib_api ~/.virtualenvs/iristest/lib/python2.7/site-packages/
# aaaand, here we go, iris!
git clone https://github.com/SciTools/iris.git
cd iris
python setup.py build
python setup.py install
# rejoice!
# pip freeze output:
# Biggus==0.12.0
# Cartopy==0.13.0
# cf-units==1.0.0
# Cython==0.23.4
# h5py==2.5.0
# Iris==1.10.0.dev0
# netCDF4==1.2.2
# numpy==1.10.2
# Pillow==3.0.0
# pyke==1.1.1
# pyshp==1.2.3
# scipy==0.16.1
# Shapely==1.5.13
# six==1.10.0
In order to install iris, using the Anaconda distribution, I did:
conda install -c conda-forge iris
and all required packages were installed at once without problems.
I have Anaconda 3 version.