Python causing: IOError: [Errno 28] No space left on device: '../results/32766.html' on disk with lots of space
Question:
I am running a Python script that is causing the above error. The unusual thing is this script is running on a different machine and is having no problems.
The difference is that on the machine that is causing the problems I am writing to an external hard drive. To make things even weirder this script has run on the problem machine and already written over 30,000 files.
Some relevant information (The code that is causing the error):
nPage = 0
while nPage != -1:
for d in data:
if len(d.contents) > 1:
if '<script' in str(d.contents):
l = str(d.contents[1])
start = l.find('http://')
end = l.find('>',start)
out = get_records.openURL(l[start:end])
print COUNT
with open('../results/'+str(COUNT)+'.html','w') as f:
f.write(out)
COUNT += 1
nPage = nextPage(mOut,False)
The directory I’m writing to:
10:32@lorax:~/econ/estc/bin$ ll ../
total 56
drwxr-xr-x 3 boincuser boincuser 4096 2011-07-31 14:29 ./
drwxr-xr-x 3 boincuser boincuser 4096 2011-07-31 14:20 ../
drwxr-xr-x 2 boincuser boincuser 4096 2011-08-09 10:38 bin/
lrwxrwxrwx 1 boincuser boincuser 47 2011-07-31 14:21 results -> /media/cavalry/server_backup/econ/estc/results//
-rw-r--r-- 1 boincuser boincuser 44759 2011-08-09 10:32 test.html
Proof there is enough space:
10:38@lorax:~/econ/estc/bin$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.0G 5.3G 3.3G 63% /
none 495M 348K 495M 1% /dev
none 500M 164K 500M 1% /dev/shm
none 500M 340K 500M 1% /var/run
none 500M 0 500M 0% /var/lock
none 9.0G 5.3G 3.3G 63% /var/lib/ureadahead/debugfs
/dev/sdc10 466G 223G 244G 48% /media/cavalry
Some things I have tried:
- Changing the path of the write to the direct location instead of going through the link
- Rebooting the machine
- Unmounting and re-mounting the drive
Answers:
The ENOSPC (“No space left on device”) error will be triggered in any situation in which the data or the metadata associated with an I/O operation can’t be written down anywhere because of lack of space. This doesn’t always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn’t space in the directory table (vfat) or there aren’t any inodes left. It roughly means “I can’t find where to write this down”.
Particularly in Python, this can happen on any write I/O operation. It can happen during f.write, but it can also happen on open, on f.flush and even on f.close. Where it happened provides a vital clue for the reason that it did – if it happened on open there wasn’t enough space to write the metadata for the entry, if it happened during f.write, f.flush or f.close there wasn’t enough disk space left or you’ve exceeded the maximum file size.
If the filesystem in the given directory is vfat you’d hit the maximum file limit at about the same time that you did. The limit is supposed to be 2^16 directory entries, but if I recall correctly some other factors can affect it (e.g. some files require more than one entry).
It would be best to avoid creating so many files in a directory. Few filesystems handle so many directory entries with ease. Unless you’re certain that your filesystem deals well with many files in a directory, you can consider another strategy (e.g. create more directories).
P.S. Also do not trust the remaining disk space – some file systems reserve some space for root and others miscalculate the free space and give you a number that just isn’t true.
It turns out the best solution for me here was to just reformat the drive. Once reformatted all these problems were no longer problems.
Try to delete the temp files
rm -r /tmp/
In my case, when I run df -i it shows me that my number of inodes are full and then I have to delete some of the small files or folder. Otherwise it will not allow us to create files or folders once inodes get full.
All you have to do is delete files or folder that has not taken up full space but is responsible for filling inodes.
run “export TEMPDIR=/someDir” where some dir is a valid directory other than /tmp.
Run this on prompt before running your python command. In my case it is “pip install rasa[spacy]” which was earlier failing.
The export command allows you to temporarily use the specified dir as temp dir.
- Show where memory is allocated
sudo du -x -h / | sort -h | tail -40
- Delete from either your
/tmp or /home/user_name/.cache folder if these are taking up a lot of memory. You can do this by running sudo rm -R /path/to/folder
Step 2 outlines fairly common folders to delete from (/tmp and /home/user_name/.cache). If you get back other results when running the first command showing you have lots of memory being used elsewhere, I advise being a bit more cautious when deleting from those locations.
I faced a similar issue. The above solutions for deleting the /tmp directory worked for me.
Instead of using the default /tmp location where the service account might not have full access (if following best practices and not using sudo to install Python packages), I moved the /tmp directory to the user’s home directory, with TMPDIR environment setting honored by pip install --user ... command.
I faced the issue above of running out space, as mentioned in the answeres above most likely due to so many files/directories being created and not actually running out of volume storage. The solution that worked for me was to delete the /home/<some_domain>/$USER/tmp directory and recreate it every time my continuous deployment pipeline ran. rm -rf /tmp
I ran into this problem when running tests.
The Django app I was running was inside a docker container, so I had to log in to the container as a root, run rm -r /tmp/ and it fixed the issue.
I have the same problem, but found the root cause is "print too frequently" at last. On my code I have some "print(…)" logic which running in loop. When I comment those code, the "No space left on device" error disappeared. Perhaps it is a Python implementation problem. You may try this solution.
For more detail, sample descriptive code will look like following:
loop:
do some logic
call "print" to check result
When your logic code running very fast, you will call "print" very frequently. Then sometimes "[Errno 28] No space left on device" error will appear.
I think this is an implementation limitation of Python "print" code, though I not read the code yet. My Python version is 3.9.7
Thanks for Community reply.
I am running a Python script that is causing the above error. The unusual thing is this script is running on a different machine and is having no problems.
The difference is that on the machine that is causing the problems I am writing to an external hard drive. To make things even weirder this script has run on the problem machine and already written over 30,000 files.
Some relevant information (The code that is causing the error):
nPage = 0
while nPage != -1:
for d in data:
if len(d.contents) > 1:
if '<script' in str(d.contents):
l = str(d.contents[1])
start = l.find('http://')
end = l.find('>',start)
out = get_records.openURL(l[start:end])
print COUNT
with open('../results/'+str(COUNT)+'.html','w') as f:
f.write(out)
COUNT += 1
nPage = nextPage(mOut,False)
The directory I’m writing to:
10:32@lorax:~/econ/estc/bin$ ll ../
total 56
drwxr-xr-x 3 boincuser boincuser 4096 2011-07-31 14:29 ./
drwxr-xr-x 3 boincuser boincuser 4096 2011-07-31 14:20 ../
drwxr-xr-x 2 boincuser boincuser 4096 2011-08-09 10:38 bin/
lrwxrwxrwx 1 boincuser boincuser 47 2011-07-31 14:21 results -> /media/cavalry/server_backup/econ/estc/results//
-rw-r--r-- 1 boincuser boincuser 44759 2011-08-09 10:32 test.html
Proof there is enough space:
10:38@lorax:~/econ/estc/bin$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.0G 5.3G 3.3G 63% /
none 495M 348K 495M 1% /dev
none 500M 164K 500M 1% /dev/shm
none 500M 340K 500M 1% /var/run
none 500M 0 500M 0% /var/lock
none 9.0G 5.3G 3.3G 63% /var/lib/ureadahead/debugfs
/dev/sdc10 466G 223G 244G 48% /media/cavalry
Some things I have tried:
- Changing the path of the write to the direct location instead of going through the link
- Rebooting the machine
- Unmounting and re-mounting the drive
The ENOSPC (“No space left on device”) error will be triggered in any situation in which the data or the metadata associated with an I/O operation can’t be written down anywhere because of lack of space. This doesn’t always mean disk space – it could mean physical disk space, logical space (e.g. maximum file length), space in a certain data structure or address space. For example you can get it if there isn’t space in the directory table (vfat) or there aren’t any inodes left. It roughly means “I can’t find where to write this down”.
Particularly in Python, this can happen on any write I/O operation. It can happen during f.write, but it can also happen on open, on f.flush and even on f.close. Where it happened provides a vital clue for the reason that it did – if it happened on open there wasn’t enough space to write the metadata for the entry, if it happened during f.write, f.flush or f.close there wasn’t enough disk space left or you’ve exceeded the maximum file size.
If the filesystem in the given directory is vfat you’d hit the maximum file limit at about the same time that you did. The limit is supposed to be 2^16 directory entries, but if I recall correctly some other factors can affect it (e.g. some files require more than one entry).
It would be best to avoid creating so many files in a directory. Few filesystems handle so many directory entries with ease. Unless you’re certain that your filesystem deals well with many files in a directory, you can consider another strategy (e.g. create more directories).
P.S. Also do not trust the remaining disk space – some file systems reserve some space for root and others miscalculate the free space and give you a number that just isn’t true.
It turns out the best solution for me here was to just reformat the drive. Once reformatted all these problems were no longer problems.
Try to delete the temp files
rm -r /tmp/
In my case, when I run df -i it shows me that my number of inodes are full and then I have to delete some of the small files or folder. Otherwise it will not allow us to create files or folders once inodes get full.
All you have to do is delete files or folder that has not taken up full space but is responsible for filling inodes.
run “export TEMPDIR=/someDir” where some dir is a valid directory other than /tmp.
Run this on prompt before running your python command. In my case it is “pip install rasa[spacy]” which was earlier failing.
The export command allows you to temporarily use the specified dir as temp dir.
- Show where memory is allocated
sudo du -x -h / | sort -h | tail -40 - Delete from either your
/tmpor/home/user_name/.cachefolder if these are taking up a lot of memory. You can do this by runningsudo rm -R /path/to/folder
Step 2 outlines fairly common folders to delete from (/tmp and /home/user_name/.cache). If you get back other results when running the first command showing you have lots of memory being used elsewhere, I advise being a bit more cautious when deleting from those locations.
I faced a similar issue. The above solutions for deleting the /tmp directory worked for me.
Instead of using the default /tmp location where the service account might not have full access (if following best practices and not using sudo to install Python packages), I moved the /tmp directory to the user’s home directory, with TMPDIR environment setting honored by pip install --user ... command.
I faced the issue above of running out space, as mentioned in the answeres above most likely due to so many files/directories being created and not actually running out of volume storage. The solution that worked for me was to delete the /home/<some_domain>/$USER/tmp directory and recreate it every time my continuous deployment pipeline ran. rm -rf /tmp
I ran into this problem when running tests.
The Django app I was running was inside a docker container, so I had to log in to the container as a root, run rm -r /tmp/ and it fixed the issue.
I have the same problem, but found the root cause is "print too frequently" at last. On my code I have some "print(…)" logic which running in loop. When I comment those code, the "No space left on device" error disappeared. Perhaps it is a Python implementation problem. You may try this solution.
For more detail, sample descriptive code will look like following:
loop:
do some logic
call "print" to check result
When your logic code running very fast, you will call "print" very frequently. Then sometimes "[Errno 28] No space left on device" error will appear.
I think this is an implementation limitation of Python "print" code, though I not read the code yet. My Python version is 3.9.7
Thanks for Community reply.