Why do Python zipapps get slower the bigger they get?
Question:
I have a python zipapp (specifically, of pdm). When running from the filesystem, it takes around 0.300s just to show the help page. However, with my zipapp, its 1s. The zipapp is around 25 MB (it includes all of the dependencies).
Getting a file in an uncompressed zip should be almost as fast as reading it since there’s an index to tell you where to go in the file to get there.
EDIT: I was able to reengineer my code to be able to be profiled with cProfile. It seems that the extra 0.6 seconds is spent with compile, but I’m not sure for what.
Answers:
I figured it out — for whatever reason, zipimport doesn’t look in __pycache__ in the zip, so it recompiles the source files each time. Replacing the files with their corresponding pyc files work fine, and speeds up execution to normal module speed.
I have a python zipapp (specifically, of pdm). When running from the filesystem, it takes around 0.300s just to show the help page. However, with my zipapp, its 1s. The zipapp is around 25 MB (it includes all of the dependencies).
Getting a file in an uncompressed zip should be almost as fast as reading it since there’s an index to tell you where to go in the file to get there.
EDIT: I was able to reengineer my code to be able to be profiled with cProfile. It seems that the extra 0.6 seconds is spent with compile, but I’m not sure for what.
I figured it out — for whatever reason, zipimport doesn’t look in __pycache__ in the zip, so it recompiles the source files each time. Replacing the files with their corresponding pyc files work fine, and speeds up execution to normal module speed.