Why are main runnable Python scripts not compiled to pyc files like modules?
Question:
I understand that when you import a module, that file is compiled into a .pyc file to make it faster? Why is the main file also not compiled to a .pyc? Does this slow things down? Would it be better to keep the main file as small as possible then, or does it not matter?
Answers:
Compiling the main script would be annoying for scripts in e.g. /usr/bin. The .pyc file is generated in the same directory, thus polluting the public location.
When a module is loaded, the py file is “byte compiled” to pyc files. The time stamp is recorded in pyc files.
This is done not to make it run faster but to load faster.
Hence, it makes sense to “byte compile” modules when you load them.
[Edit : To include notes, references]
From PEP 3147 on “Byte code
compilation”:
CPython compiles its source code into “byte code”, and for performance reasons,
it caches this byte code on the file system whenever the source file has changes.
This makes loading of Python modules much faster because the compilation phase
can be bypassed. When your source file is foo.py, CPython caches the byte
code in a foo.pyc file right next to the source.
How byte code compiled files are
tracked with respect to Python version
and “py” file changes:
It also inserts a magic number in the compiled byte code “.pyc” files.
This changes whenever Python changes the byte code format, usually in major releases.
This ensures that pyc files built for previous versions of the VM won’t cause problems.
The timestamp is used to make sure that the pyc file match the py file that was
used to create it. When either the magic number or timestamp do not match,
the py file is recompiled and a new pyc file is written.
“pyc” files are not compatible across Python major releases. When Python finds a pyc
file with a non-matching magic number, it falls back to the slower process of
recompiling the source.
Thats the reason, if you simply distribute the “.pyc” files compiled for the same platform will not work any more, if the python version changes.
In Nutshell
If there is a byte compiled file “.pyc” and it’s timestamp indicates that it is recent then it will be loaded up other wise python will fallback on the slower approach of loading the “.py” files. The execution performance of the “.py” file is not affected but the loading of the “.pyc” files is faster than “.py” files.
Consider executing a.py which imports b.py
Typical total performance = loading time (A.py) + execution time (A.py) +
loading time (B.py) + execution time (B.py)
Since loading time (B.pyc) < loading time (B.py)
You should see a better performance by using the byte compiled "pyc" files.
That said, if you have a large script file X.py, modularizing it and moving contents to other modules results in taking advantage of lower load time for byte code compiled file.
Another inference is that modules tend to be more stable than the script or the main file. Hence it is not byte compiled at all.
References
I understand that when you import a module, that file is compiled into a .pyc file to make it faster? Why is the main file also not compiled to a .pyc? Does this slow things down? Would it be better to keep the main file as small as possible then, or does it not matter?
Compiling the main script would be annoying for scripts in e.g. /usr/bin. The .pyc file is generated in the same directory, thus polluting the public location.
When a module is loaded, the py file is “byte compiled” to pyc files. The time stamp is recorded in pyc files.
This is done not to make it run faster but to load faster.
Hence, it makes sense to “byte compile” modules when you load them.
[Edit : To include notes, references]
From PEP 3147 on “Byte code
compilation”:
CPython compiles its source code into “byte code”, and for performance reasons,
it caches this byte code on the file system whenever the source file has changes.
This makes loading of Python modules much faster because the compilation phase
can be bypassed. When your source file is foo.py, CPython caches the byte
code in a foo.pyc file right next to the source.
How byte code compiled files are
tracked with respect to Python version
and “py” file changes:
It also inserts a magic number in the compiled byte code “.pyc” files.
This changes whenever Python changes the byte code format, usually in major releases.
This ensures that pyc files built for previous versions of the VM won’t cause problems.
The timestamp is used to make sure that the pyc file match the py file that was
used to create it. When either the magic number or timestamp do not match,
the py file is recompiled and a new pyc file is written.
“pyc” files are not compatible across Python major releases. When Python finds a pyc
file with a non-matching magic number, it falls back to the slower process of
recompiling the source.
Thats the reason, if you simply distribute the “.pyc” files compiled for the same platform will not work any more, if the python version changes.
In Nutshell
If there is a byte compiled file “.pyc” and it’s timestamp indicates that it is recent then it will be loaded up other wise python will fallback on the slower approach of loading the “.py” files. The execution performance of the “.py” file is not affected but the loading of the “.pyc” files is faster than “.py” files.
Consider executing a.py which imports b.py
Typical total performance = loading time (A.py) + execution time (A.py) +
loading time (B.py) + execution time (B.py)
Since loading time (B.pyc) < loading time (B.py)
You should see a better performance by using the byte compiled "pyc" files.
That said, if you have a large script file X.py, modularizing it and moving contents to other modules results in taking advantage of lower load time for byte code compiled file.
Another inference is that modules tend to be more stable than the script or the main file. Hence it is not byte compiled at all.
References