Way to have compiled python files in a separate folder?
Question:
Is it possible to have Python save the .pyc files to a separate folder location that is in sys.path?
/code
foo.py
foo.pyc
bar.py
bar.pyc
To:
/code
foo.py
bar.py
/code_compiled
foo.pyc
bar.pyc
I would like this because I feel it’d be more organized. Thanks for any help you can give me.
Answers:
Update:
In Python 3.8 -X pycache_prefix=PATH command-line option enables writing .pyc files to a parallel tree rooted at the given directory instead of to the code tree. See $PYTHONPYCACHEPREFIX envvarcredits: @RobertT’ answer
The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ [since Python 3.2] subdirectories).
To turn off caching the compiled Python bytecode, -B may be set, then Python won’t try to write .pyc files on the import of source modules. See $PYTHONDONTWRITEBYTECODE envvarcredits: @Maleev’s answer
Old [Python 2] answer:
There is PEP 304: Controlling Generation of Bytecode Files. Its status is Withdrawn and corresponding patch rejected. Therefore there might be no direct way to do it.
If you don’t need source code then you may just delete *.py files. *.pyc files can be used as is or packed in an egg.
“I feel it’d be more organized” Why? How? What are you trying to accomplish?
The point of saving the compiler output is to save a tiny bit of load time when the module gets imported. Why make this more complex? If you don’t like the .pyc’s, then run a “delete all the .pyc’s” script periodically.
They aren’t essential; they’re helpful. Why turn off that help?
This isn’t C, C++ or Java where the resulting objects are essential. This is just a cache that Python happens to use. We mark them as “ignored” in Subversion so they don’t accidentally wind up getting checked in.
I agree, distributing your code as an egg is a great way to keep it organized. What could be more organized than a single-file containing all of the code and meta-data you would ever need. Changing the way the bytecode compiler works is only going to cause confusion.
If you really do not like the location of those pyc files, an alternative is to run from a read-only folder. Since python will not be able to write, no pyc files ever get made. The hit you take is that every python file will have to be re-compiled as soon as it is loaded, regardless of whether you have changed it or not. That means your start-up time will be a lot worse.
I disagree. The reasons are wrong or at least not well formulated; but the direction is valid. There are good reasons for being able to segregate source code from compiled objects. Here are a few of them (all of them I have run into at one point or another):
- embedded device reading off a ROM, but able to use an in memory filesystem on RAM.
- multi-os dev environment means sharing (with samba/nfs/whatever) my working directory and building on multiple platforms.
- commercial company wishes to only distribute pyc to protect the IP
- easily run test suite for multiple versions of python using the same working directory
- more easily clean up transitional files (rm -rf $OBJECT_DIR as opposed to find . -name ‘*.pyc’ -exec rm -f {} 😉
There are workarounds for all these problems, BUT they are mostly workarounds NOT solutions. The proper solution in most of these cases would be for the software to accept an alternative location for storing and lookup of these transitional files.
If you’re willing to sacrifice bytecode generation altogether for it, there’s a command line flag:
python -B file_that_imports_others.py
Can be put into IDE’s build/run preferences
There is ongoing pep that will enable building bytecode to magic directory.
Basically all python files will be compiled to directory __pythoncache__.
Since Python 3.2 has been implemented PEP 3147: this means that all .pyc files are generated inside a __pycache__ directory (there will be a __pycache__ directory for each directory where you have Python files, and it will hold .pyc files for each version of Python used on the sources)
In the dark and ancient days of 2003, PEP 304 came forth to challenge this problem. Its patch was found wanting. Environment variable platform dependencies and version skews ripped it to shreds and left its bits scattered across the wastelands.
After years of suffering, a new challenger rose in the last days of 2009. Barry Warsaw summoned PEP 3147 and sent it to do battle, wielding a simple weapon with skill. The PEP crushed the cluttering PYC files, silenced the waring Unladen Swallow and CPython interpreter each trying to argue its PYC file should be triumphant, and allowed Python to rest easy with its dead ghosts occasionally running in the dead of night. PEP 3147 was found worthy by the dictator and was knighted into the official roles in the days of 3.2.
As of 3.2, Python stores a module’s PYC files in __pycache__ under the module’s directory. Each PYC file contains the name and version of the interpreter, e.g., __pycache__/foo.cpython-33.pyc. You might also have a __pycache__/foo.cpython-32.pyc compiled by an earlier version of Python. The right magic happens: the correct one is used and recompiled if out of sync with the source code. At runtime, look at the module’s mymodule.__cached__ for the pyc filename and parse it with imp.get_tag(). See the What’s New section for more information.
TL;DR – Just works in Python 3.2 and above. Poor hacks substitute for versions before that.
And only almost ten years later, Python 3.8 finally provides support for keeping bytecode in separate parallel filesystem tree by setting environment variable PYTHONPYCACHEPREFIX or using -X pycache_prefix=PATH argument (official doc here).
For Python 3.8 or higher:
The PYTHONPYCACHEPREFIX setting (also available as -X pycache_prefix) configures the implicit bytecode cache to use a separate filesystem tree, rather than the default __pycache__ subdirectories within each source directory.
The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ subdirectories).
Is it possible to have Python save the .pyc files to a separate folder location that is in sys.path?
/code
foo.py
foo.pyc
bar.py
bar.pyc
To:
/code
foo.py
bar.py
/code_compiled
foo.pyc
bar.pyc
I would like this because I feel it’d be more organized. Thanks for any help you can give me.
Update:
In Python 3.8 -X pycache_prefix=PATH command-line option enables writing .pyc files to a parallel tree rooted at the given directory instead of to the code tree. See $PYTHONPYCACHEPREFIX envvarcredits: @RobertT’ answer
The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ [since Python 3.2] subdirectories).
To turn off caching the compiled Python bytecode, -B may be set, then Python won’t try to write .pyc files on the import of source modules. See $PYTHONDONTWRITEBYTECODE envvarcredits: @Maleev’s answer
Old [Python 2] answer:
There is PEP 304: Controlling Generation of Bytecode Files. Its status is Withdrawn and corresponding patch rejected. Therefore there might be no direct way to do it.
If you don’t need source code then you may just delete *.py files. *.pyc files can be used as is or packed in an egg.
“I feel it’d be more organized” Why? How? What are you trying to accomplish?
The point of saving the compiler output is to save a tiny bit of load time when the module gets imported. Why make this more complex? If you don’t like the .pyc’s, then run a “delete all the .pyc’s” script periodically.
They aren’t essential; they’re helpful. Why turn off that help?
This isn’t C, C++ or Java where the resulting objects are essential. This is just a cache that Python happens to use. We mark them as “ignored” in Subversion so they don’t accidentally wind up getting checked in.
I agree, distributing your code as an egg is a great way to keep it organized. What could be more organized than a single-file containing all of the code and meta-data you would ever need. Changing the way the bytecode compiler works is only going to cause confusion.
If you really do not like the location of those pyc files, an alternative is to run from a read-only folder. Since python will not be able to write, no pyc files ever get made. The hit you take is that every python file will have to be re-compiled as soon as it is loaded, regardless of whether you have changed it or not. That means your start-up time will be a lot worse.
I disagree. The reasons are wrong or at least not well formulated; but the direction is valid. There are good reasons for being able to segregate source code from compiled objects. Here are a few of them (all of them I have run into at one point or another):
- embedded device reading off a ROM, but able to use an in memory filesystem on RAM.
- multi-os dev environment means sharing (with samba/nfs/whatever) my working directory and building on multiple platforms.
- commercial company wishes to only distribute pyc to protect the IP
- easily run test suite for multiple versions of python using the same working directory
- more easily clean up transitional files (rm -rf $OBJECT_DIR as opposed to find . -name ‘*.pyc’ -exec rm -f {} 😉
There are workarounds for all these problems, BUT they are mostly workarounds NOT solutions. The proper solution in most of these cases would be for the software to accept an alternative location for storing and lookup of these transitional files.
If you’re willing to sacrifice bytecode generation altogether for it, there’s a command line flag:
python -B file_that_imports_others.py
Can be put into IDE’s build/run preferences
There is ongoing pep that will enable building bytecode to magic directory.
Basically all python files will be compiled to directory __pythoncache__.
Since Python 3.2 has been implemented PEP 3147: this means that all .pyc files are generated inside a __pycache__ directory (there will be a __pycache__ directory for each directory where you have Python files, and it will hold .pyc files for each version of Python used on the sources)
In the dark and ancient days of 2003, PEP 304 came forth to challenge this problem. Its patch was found wanting. Environment variable platform dependencies and version skews ripped it to shreds and left its bits scattered across the wastelands.
After years of suffering, a new challenger rose in the last days of 2009. Barry Warsaw summoned PEP 3147 and sent it to do battle, wielding a simple weapon with skill. The PEP crushed the cluttering PYC files, silenced the waring Unladen Swallow and CPython interpreter each trying to argue its PYC file should be triumphant, and allowed Python to rest easy with its dead ghosts occasionally running in the dead of night. PEP 3147 was found worthy by the dictator and was knighted into the official roles in the days of 3.2.
As of 3.2, Python stores a module’s PYC files in __pycache__ under the module’s directory. Each PYC file contains the name and version of the interpreter, e.g., __pycache__/foo.cpython-33.pyc. You might also have a __pycache__/foo.cpython-32.pyc compiled by an earlier version of Python. The right magic happens: the correct one is used and recompiled if out of sync with the source code. At runtime, look at the module’s mymodule.__cached__ for the pyc filename and parse it with imp.get_tag(). See the What’s New section for more information.
TL;DR – Just works in Python 3.2 and above. Poor hacks substitute for versions before that.
And only almost ten years later, Python 3.8 finally provides support for keeping bytecode in separate parallel filesystem tree by setting environment variable PYTHONPYCACHEPREFIX or using -X pycache_prefix=PATH argument (official doc here).
For Python 3.8 or higher:
The PYTHONPYCACHEPREFIX setting (also available as -X pycache_prefix) configures the implicit bytecode cache to use a separate filesystem tree, rather than the default __pycache__ subdirectories within each source directory.
The location of the cache is reported in sys.pycache_prefix (None indicates the default location in __pycache__ subdirectories).