Is there a faster way to compress files/make an archive in Python?
Question:
I’m making archive of a folder that has around ~1 GB of files in it. It takes like 1 or 2 minutes but I want it to be faster.
I am making a UI app in Python that allows you to ZIP files (it’s a project, I know stuff like 7Zip exists lol), and I am using a folder with ~1GB of files in it, like I said. The program won’t accept files over 5GB because of the speed.
Here is how I compress/archive the folder:
shutil.make_archive('files_archive', 'zip', 'folder_to_archive')
I know I can change the format but I’m not sure which one is fast and can compress well.
So, my end goal is to compress files fast, whilst also reducing size.
Answers:
shutil.make_archive offers zip, tar, gztar, bztar or xztar as archive formats.
tar is uncompressed so will be fastest, but uncompressed.
If you want to have faster compression than the built in formats, you will need an external command.
A common fast compression algorithm is LZ4.
I recommend you look into https://morotti.github.io/lzbench-web/ for various compression algorithms, and their compression ratio over speed.
But you will likely have to benchmark on a sample of own data.
I’m making archive of a folder that has around ~1 GB of files in it. It takes like 1 or 2 minutes but I want it to be faster.
I am making a UI app in Python that allows you to ZIP files (it’s a project, I know stuff like 7Zip exists lol), and I am using a folder with ~1GB of files in it, like I said. The program won’t accept files over 5GB because of the speed.
Here is how I compress/archive the folder:
shutil.make_archive('files_archive', 'zip', 'folder_to_archive')
I know I can change the format but I’m not sure which one is fast and can compress well.
So, my end goal is to compress files fast, whilst also reducing size.
shutil.make_archive offers zip, tar, gztar, bztar or xztar as archive formats.
tar is uncompressed so will be fastest, but uncompressed.
If you want to have faster compression than the built in formats, you will need an external command.
A common fast compression algorithm is LZ4.
I recommend you look into https://morotti.github.io/lzbench-web/ for various compression algorithms, and their compression ratio over speed.
But you will likely have to benchmark on a sample of own data.