Is there a true way to gradually iterate over a large directory in python?
Question:
I don’t quite understand which function should I use to gradually iterate over files in a directory (non-recursively) without precaching them all in a one huge list.
I checked the documentation of os.scandir(), but it doesn’t explicitly specify whether or not it loads the entire list into memory at once.
I’ve also checked the documentation on .iglob(), but it was later revealed that it does store all the items in memory…
Answers:
According to PEP 471 – os.scandir() function, os.scandir will do what you want:
It returns a generator instead of a list, so that scandir acts as a
true iterator instead of returning the full list immediately.
It uses the system calls FindFirstFile / FindNextFile on Windows and readdir on POSIX systems to avoid loading the entire list of files into memory.
I don’t quite understand which function should I use to gradually iterate over files in a directory (non-recursively) without precaching them all in a one huge list.
I checked the documentation of os.scandir(), but it doesn’t explicitly specify whether or not it loads the entire list into memory at once.
I’ve also checked the documentation on .iglob(), but it was later revealed that it does store all the items in memory…
According to PEP 471 – os.scandir() function, os.scandir will do what you want:
It returns a generator instead of a list, so that scandir acts as a
true iterator instead of returning the full list immediately.
It uses the system calls FindFirstFile / FindNextFile on Windows and readdir on POSIX systems to avoid loading the entire list of files into memory.