Why does saving `to_netcdf` without `encoding=` change some values to `nan`?

I’m struggling to understand a problem in my code when saving a xarray.DataSet as netCDF. The file does not contain any nan values. However after saving and loading it suddenly does for one value:

1. Before: no nan in the original data before saving:

   
   > ds.where(lambda x: x.isnull(), drop=True).coords
   
   Coordinates:
     * x        (x) float64 
     * y        (y) float64 
     * time     (time) datetime64[ns] 
       lon      (x) float64 
       lat      (y) float64 
   

2. Saving:

   > ds.to_netcdf("manual_save.nc")
   

3. Loading: Now a nan appears for a single data entry. Only this entry is affected. The effect is reproducible.

   > xr.open_dataset("manual_save.nc").where(lambda x: x.isnull(), drop=True).coords
   
   Coordinates:
     * x        (x) float64 -3.5
     * y        (y) float64 57.0
     * time     (time) datetime64[ns] 2023-02-01
       lon      (x) float64 -3.5
       lat      (y) float64 57.0
   

I don’t understand why this is happening, can someone explain and offer a good solution?

More details

1. Here’s the value before and after saving+loading of the affected entry:

   # Before saving+loading
   > ds["soil temperature"].sel(x=-3.5, y=57, time="2023-02-01 00:00").load()
   
   <xarray.DataArray 'soil temperature' ()>
   array(275.88766, dtype=float32)
   Coordinates:
       x        float64 -3.5
       y        float64 57.0
       time     datetime64[ns] 2023-02-01
       lon      float64 -3.5
       lat      float64 57.0
   Attributes:
       units:      K
       long_name:  Soil temperature level 4
       module:     era5
       feature:    temperature
   
   # After saving+loading
   > xr.open_dataset("manual_save.nc")["soil temperature"].sel(x=-3.5, y=57, time="2023-02-01 00:00").load()
   
   <xarray.DataArray 'soil temperature' ()>
   array(nan, dtype=float32)
   Coordinates:
       x        float64 -3.5
       y        float64 57.0
       time     datetime64[ns] 2023-02-01
       lon      float64 -3.5
       lat      float64 57.0
   Attributes:
       units:      K
       long_name:  Soil temperature level 4
       module:     era5
       feature:    temperature
   

2. Before saving the data is represented as a dask.array in xarray, requiring the .load() to show the value. Without .load() it looks like this before saving:

   > ds["soil temperature"].sel(x=-3.5, y=57, time="2023-02-01 00:00")
   
   <xarray.DataArray 'soil temperature' ()>
   dask.array<getitem, shape=(), dtype=float32, chunksize=(), chunktype=numpy.ndarray>
   Coordinates:
       x        float64 -3.5
       y        float64 57.0
       time     datetime64[ns] 2023-02-01
       lon      float64 -3.5
       lat      float64 57.0
   Attributes:
       units:      K
       long_name:  Soil temperature level 4
       module:     era5
       feature:    temperature
   

3. Here’s a peak at the full xarray.DataSet. No other entries are affected by the problem:

   > ds
   
   <xarray.Dataset>
   Dimensions:           (x: 23, y: 25, time: 48)
   Coordinates:
     * x                 (x) float64 -4.0 -3.75 -3.5 -3.25 ... 0.75 1.0 1.25 1.5
     * y                 (y) float64 56.0 56.25 56.5 56.75 ... 61.5 61.75 62.0
     * time              (time) datetime64[ns] 2023-01-31 ... 2023-02-01T23:00:00
       lon               (x) float64 -4.0 -3.75 -3.5 -3.25 ... 0.75 1.0 1.25 1.5
       lat               (y) float64 56.0 56.25 56.5 56.75 ... 61.5 61.75 62.0
   Data variables:
       temperature       (time, y, x) float32 dask.array<chunksize=(24, 25, 23), meta=np.ndarray>
       soil temperature  (time, y, x) float32 dask.array<chunksize=(24, 25, 23), meta=np.ndarray>
   Attributes:
       module:             era5
       prepared_features:  ['temperature']
       chunksize_time:     100
       Conventions:        CF-1.6
       history:            2023-03-13 09:15:56 GMT by grib_to_netcdf-2.25.1: /op...
   

4. I can workaround the issue by specifying an compression with zlib via encoding:

   
   > ds.to_netcdf("manual_save_with_zlib.nc", encoding={'soil temperature': {'zlib': True, 'complevel': 1}})
   > xr.open_dataset("manual_save_with_zlib.nc")["soil temperature"].sel(x=-3.5, y=57, time="2023-02-01 00:00").load()
   
   <xarray.DataArray 'soil temperature' ()>
   array(275.88766, dtype=float32)
   Coordinates:
       x        float64 -3.5
       y        float64 57.0
       time     datetime64[ns] 2023-02-01
       lon      float64 -3.5
       lat      float64 57.0
   Attributes:
       units:      K
       long_name:  Soil temperature level 4
       module:     era5
       feature:    temperature
   
   

5. The DataSet is created quite deep inside the code of a library of ours from the online API of ERA5, so I don’t know how to create a MWE to share for this issue. The API access and retrieved data all seem to work fine as always.

6. (edit) As suggested by psalt I tried .compute() before saving and explicitly specifying compute=True while saving to remove this potential dask stumbling block. Neither change the result, after loading the nan value still exist. Here’s what I did:

   > ds.compute().to_netcdf("manual_save_pre-compute.nc")
   > ds.to_netcdf("manual_save-and-compute.nc", compute=True)
   

7. (edit) I also tried saving to zarr but without any success either. The same problem occurs there after loading.

8. (out of date)

! (edit) I’m sharing the affected DataSet as pickle because all standard methods from xarray interfere with the problem. If you unpickle the version and then save the DataSet as described above, you can reproduce the problem. You can download the pickle file here.
!
>! python >! > import pickle >! >! # Code used for creating the pickle >! > f = open("manual_save.pickle", "wb") >! > pickle.dump(ds, f, protocol=pickle.HIGHEST_PROTOCOL) >! > f.close() >! >! # Code for unpickling >! with open("manual_save.pickle", "rb") as f: >! ds = pickle.load(f)~ >!

9. (edit) I’ve managed to track down the error to an instable netCDF file. You can download the file here. Tested with xarray=2023.2.0 the following code seems to create a nan value out of thin air:

   import xarray as xr
   
   ds = xr.open_mfdataset("instable-datafile.nc")
   display("This contains no nan values", ds["t2m"].values)
   
   ds.to_netcdf("collapsed-datafile.nc")
   display("This contains nan values", xr.open_dataset("collapsed-datafile.nc")["t2m"].values)
   
   # Output
   
   'This contains no nan values'
   
   array([[[278.03146, 278.4846 ],
           [278.50998, 278.6799 ]],
   
           [[277.91476, 278.4109 ],
           [278.36594, 278.571  ]]], dtype=float32)
   
   'This contains nan values'
   
   array([[[278.03146, 278.4846 ],
           [278.50998, 278.6799 ]],
   
           [[      nan, 278.4109 ],
           [278.36594, 278.571  ]]], dtype=float32)
   

I’m happy to provide more information. Just let me know.