OSError: [Errno 24] Too many open files: '/home/ec2-user/car/1016780737.jpg'

Question

I am doing predictions on 100K images using following yolov8 code:

model = YOLO(self.weightpath)
src_dir = self.src_Dir
src_dir = src_dir+ '*'        
img_list = glob.glob(src_dir)
results = model(img_list, max_det = 1)

I am getting following error:

Traceback (most recent call last):
  File "/home/ec2-user/Deployment/main/Quality_check.py", line 63, in <module>
  File "/home/ec2-user/Deployment/utils/save_img_bins.py", line 19, in run
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/engine/model.py", line 102, in __call__
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/engine/model.py", line 202, in predict
  File "/home/ec2-user/.local/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/engine/predictor.py", line 116, in __call__
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/engine/predictor.py", line 147, in stream_inference
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/engine/predictor.py", line 135, in setup_source
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/data/build.py", line 164, in load_inference_source
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/data/build.py", line 148, in check_source
  File "/home/ec2-user/.local/lib/python3.7/site-packages/ultralytics/yolo/data/dataloaders/stream_loaders.py", line 339, in autocast_list
  File "/home/ec2-user/.local/lib/python3.7/site-packages/PIL/Image.py", line 3227, in open
OSError: [Errno 24] Too many open files: '/home/ec2-user/car/1016780737.jpg'

Asked By: Hitesh

||

Source

Answer 1

Make sure your program is expected to open that many files. Open files are a limited resource there is a limit set for a reason. Having said that, the default limits may be probably too low for some applications. Luckily, you can change the limit easily. Here is an article, which explains all the details, but in general

cat /proc/sys/fs/file-max system wide limit

ulimit -n limit per process

ulimit -u limit for the current user

ulimit -n 2048 – set the process limit to 2048

On a standard unix-based system the user can modify the limits to some extend even without admin permissions (soft limits, see the article), just again – be careful. Too many files, may make your system slow down or even fail. If your model needs to open all the files at once or does not close them after using them, maybe fix the algorithm first.

https://www.howtogeek.com/805629/too-many-open-files-linux/#:~:text=If%20you%20see%20the%20%22Too,by%20editing%20systemd%20configuration%20files.

Answered By: petrch

OSError: [Errno 24] Too many open files: '/home/ec2-user/car/1016780737.jpg'

Question:

Answers: