PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

Question

I am a beginner at PyTorch and I am just trying out some examples on this webpage. But I can’t seem to get the ‘super_resolution’ program running due to this error:

RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

I searched the Internet and found that some people suggest setting num_workers to 0. But if I do that, the program tells me that I am running out of memory (either with CPU or GPU):

RuntimeError: [enforce fail at ..c10coreCPUAllocator.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 9663676416 bytes. Buy new RAM!

or

RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 0 bytes free; 2.03 GiB reserved in total by PyTorch)

How do I fix this?

I am using python 3.8 on Win10(64bit) and pytorch 1.4.0.

More complete error messages (--cuda means using GPU, --threads x means passing x to the num_worker parameter):

with command line arguments --upscale_factor 1 --cuda

  File "E:Python38libsite-packagestorchutilsdatadataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "E:Python38libmultiprocessingqueues.py", line 108, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "Z:super_resolutionmain.py", line 81, in <module>
    train(epoch)
  File "Z:super_resolutionmain.py", line 48, in train
    for iteration, batch in enumerate(training_data_loader, 1):
  File "E:Python38libsite-packagestorchutilsdatadataloader.py", line 345, in __next__
    data = self._next_data()
  File "E:Python38libsite-packagestorchutilsdatadataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "E:Python38libsite-packagestorchutilsdatadataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "E:Python38libsite-packagestorchutilsdatadataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 16596, 9376, 12756, 9844) exited unexpectedly

with command line arguments --upscale_factor 1 --cuda --threads 0

  File "Z:super_resolutionmain.py", line 81, in <module>
    train(epoch)
  File "Z:super_resolutionmain.py", line 52, in train
    loss = criterion(model(input), target)
  File "E:Python38libsite-packagestorchnnmodulesmodule.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "Z:super_resolutionmodel.py", line 21, in forward
    x = self.relu(self.conv2(x))
  File "E:Python38libsite-packagestorchnnmodulesmodule.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "E:Python38libsite-packagestorchnnmodulesconv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "E:Python38libsite-packagestorchnnmodulesconv.py", line 341, in conv2d_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 1024.00 MiB (GPU 0; 4.00 GiB total capacity; 2.03 GiB already allocated; 954.35 MiB free; 2.03 GiB reserved in total by PyTorch)

Asked By: ihdv

||

Source

Answer 1

There is no “complete” solve for GPU out of memory errors, but there are quite a few things you can do to relieve the memory demand. Also, make sure that you are not passing the trainset and testset to the GPU at the same time!

Decrease batch size to 1
Decrease the dimensionality of the fully-connected layers (they are the most memory-intensive)
(Image data) Apply centre cropping
(Image data) Transform RGB data to greyscale
(Text data) Truncate input at n chars (which probably won’t help that much)

Alternatively, you can try running on Google Colaboratory (12 hour usage limit on K80 GPU) and Next Journal, both of which provide up to 12GB for use, free of charge. Worst case scenario, you might have to conduct training on your CPU. Hope this helps!

Answered By: ccl

Answer 2

Restart your system for the GPU to regain its memory. Save all the work and restart your System.

Answered By: Gokhulan Damodaran

Answer 3

I tried to fine-tuning it using different combinations. The solution for me is on batch_size = 1 and n_of_jobs=8

Answered By: Iasonas Christoulakis

Answer 4

Reduce number of workers, -- threads x in your case.

Answered By: Chris Cao

Answer 5

This is the solution that worked for me. it may work for other Windows users.
Just remove/comment the num workers to disable parallel loads

Answered By: Aneesh Cherian K

Answer 6

On windows Aneesh Cherian’s solution works well for notebooks (IPython). But if you want to use num_workers>0 you should avoid interpreters like IPython and put the dataload in if __name__ == '__main__:. Also, with persistent_workers=True the dataload appears to be faster on windows if num_workers>0.

More information can be found in this thread: https://github.com/pytorch/pytorch/issues/12831

Answered By: gaspar

Answer 7

As the accepted response states. There is no explicit solution. However, in my case, I had to resize all images as the images were large and model huge. You can refer to this post for resizing: https://stackoverflow.com/a/73798986/16599761

Answered By: Timilehin A.

Answer 8

I was working with mmaction trainer when this error showed up. What worked for me was:

cfg.data['workers_per_gpu']=0

Where cfg is the configuration, for training.

Answered By: Om Rastogi

PyTorch RuntimeError: DataLoader worker (pid(s) 15332) exited unexpectedly

Question:

Answers: