importing dask_cuda results in parse_memory_limit error

Question:

I’m trying to import dask_cuda as the examples:

from dask_cuda import LocalCUDACluster
from dask.distributed import Client

But I receive the following error:

ImportError                               Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 from dask_cuda import LocalCUDACluster

File ~/miniconda3/lib/python3.8/site-packages/dask_cuda/__init__.py:5, in <module>
      2 import dask.dataframe.shuffle
      4 from ._version import get_versions
----> 5 from .cuda_worker import CUDAWorker
      6 from .explicit_comms.dataframe.shuffle import get_rearrange_by_column_tasks_wrapper
      7 from .local_cuda_cluster import LocalCUDACluster

File ~/miniconda3/lib/python3.8/site-packages/dask_cuda/cuda_worker.py:19, in <module>
     14 from distributed.proctitle import (
     15     enable_proctitle_on_children,
     16     enable_proctitle_on_current,
     17 )
     18 from distributed.utils import parse_bytes
---> 19 from distributed.worker import parse_memory_limit
     21 from .device_host_file import DeviceHostFile
     22 from .initialize import initialize

ImportError: cannot import name 'parse_memory_limit' from 'distributed.worker' (/home/shambakey1/miniconda3/lib/python3.8/site-packages/distributed/worker.py)

I googled this error, but couldn’t find anything related to it. I thought it was a DASK problem, so I re-tried with only from dask.distributed import Client, and it worked. So, I guess this error is related to dask_cuda, but I don’t know how.

The version I have are:

  1. dask-cuda -> 22.6.0
  2. dask -> 2021.12.0
  3. distributed -> 2021.12.0
  4. Ubuntu 20.04

Regards

Asked By: shambakey1

||

Answers:

Your dask version is rather old, the parse_memory_limit was added fairly recently in 2022, see this PR.

I would recommend not pinning dask and let conda find the most suitable version. If that’s not an option, then dask=2022.7.0 works for me with dask-cuda=2022.6.0.

Answered By: SultanOrazbayev
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.