What is the effect of using `python -m pip` instead of just `pip`?
Question:
When I use python -m pip install <package>, how is that different from using just pip install <package>? Similarly, why would I write python -m pip install --upgrade pip to upgrade Pip, instead of just pip install --upgrade pip?
Answers:
If you type python --help
You get
// More flags above
-m mod : run library module as a script (terminates option list)
// and more flags below
A great many things in a terminal will show you how to use it if you either use command --help or man command
The -m stands for module-name.
From Command line and environment:
python [-bBdEhiIOqsSuvVWx?] [-c command | -m module-name | script | – ] [args]
From Python Docs:
Since the argument is a module name, you must not give a file extension (.py). The module-name should be a valid Python module name, but the implementation may not always enforce this (e.g. it may allow you to use a name that includes a hyphen).
Package names are also permitted. When a package name is supplied instead of a normal module, the interpreter will execute <pkg>.__main__ as the main module. This behaviour is deliberately similar to the handling of directories and zipfiles that are passed to the interpreter as the script argument.
When -m is used with a python statement on the command line, followed by a <module_name>, then it enables the module to be executed as an executable file.
You can refer to python docs for the same, or run python --help
If you have multiple versions of python installed and you want to upgrade pip pip install --upgrade pip how do you know which python version will be affected? it depends on path variable for the shell. You might also get warning in this case. To avoid this confusion use -m then it looks in variable sys.path. This is another advantage of -m.
# importing module
import sys
# printing all directories for
# interpreter to search
sys.path
Consider the following scenario.
You have three versions of Python installed:
- Python 3.7
- Python 3.8
- Python 3.9
Your "default" version is 3.8. It’s the first one appearing in your path. Therefore, when you type python3 (Linux or Mac) or python (Windows) in a shell you will start a 3.8 interpreter because that’s the first Python executable that is found when traversing your path.
Suppose you are then starting a new project where you want to use Python 3.9. You create a virtual environment called .venv and activate it.
python3.9 -m venv .venv # "py -3.9" on Windows
source .venv/bin/activate # ".venvScriptsactivate" on Windows
We now have the virtual environment activated using Python 3.9. Typing python in a shell starts the 3.9 interpreter.
BUT, if you type
pip install <some-package>
Then what version of pip is used? Is it the pip for the default version, i.e. Python 3.8, or the Python version within the virtual environment?
An easy way to get around that ambiguity is simply to use
python -m pip install <some-package>
The -m flag makes sure that you are using the pip that’s tied to the active Python executable.
It’s good practice to always use -m, even if you have just one global version of Python installed from which you create virtual environments.
Re. path
The so-called path is a list of directories where your system searches for executables. When you type a command, like python, this list is traversed from the first directory to the last, searching for a filename that matches the command you typed.
If the filename/command is found, the matched file gets executed without taking into account potential later matches. If no match occurs, you get a Command not found or a variation thereof. This behavior is by design.
On UNIX systems the path environment variable is called $PATH, while on Windows systems it’s referred to as %PATH%
More general comments about the -m-flag (Dec. 2022)
Most people viewing this will likely want the explanation given above with pip. In a more general sense though, when using python -m some_module, the -m flag makes Python execute some_module as a script. This is stated in the docs, but might be difficult to understand without some baseline knowledge.
What does it mean to "run as a script"?
In Python, a module some_module is typically imported into another Python file with an import some_module statement at the top of the importing file. This enables the use of functions, classes, and variables defined in some_module inside the importing file.
To execute some_module as a script instead of importing it, you would define an if __name__ == "__main__" block inside the file. This block gets executed when running python some_module.py on the command line. This is useful because you do not want this code block to run when importing into other files, but you do want it to run when invoked from the command line.
For modules inside your project, this script/module construct should run as is, because Python will find the module from your working directory when running from the terminal:
python some_module.py
But for modules that are part of Python’s Standard Library, this will not work. The example from the Python docs uses timeit (pip works the same):
python3 timeit -s 'print("hello")' # 'python timeit.py ...' fails as well
This returns the error: "python: can't open file '/home/<username>/timeit': [Errno 2] No such file or directory"
Adding the -m flag tells Python to look in path for timeit.py and execute the if __name__ == "__main__" clause from the file.
python3 -m timeit -s 'print("hello")'
This works as expected.
The source for the if __name__ == "__main__" block for the timeit module can be found here.
This is actually an interesting question, so let’s explore pep 338 linked by @jedwards in the top comment.
The -m flag originally served a simpler purpose- to convert a module name into a script name. In python 2.4, the behavior was:
the command line is effectively reinterpreted from python <options> -m
<module> <args> to python <options> <filename> <args>.
This doesn’t seem very useful, but that’s what it did back then. Pep 338 extends this behavior further.
The semantics proposed are fairly simple [sic]: if -m is used to
execute a module the PEP 302 import mechanisms are used to locate the
module and retrieve its compiled code, before executing the module in
accordance with the semantics for a top-level module.
It goes on to explain further that python will identify the package the module is from, import that package using the standard process, and run the module. From what I understand calling "python3 -m package.module" is the same as calling:
python3
from package import module
The -m flag will run the module as __file__ and not __main__. It will also insert the local directory into sys.path instead of the script’s directory. It therefore breaks relative imports, although this was not intentional, and the authors therefore recommend to always use absolute imports. It also depends on how you call it- "python3 -m package.module" is not the same as "python3 -m module".
In theory it’s simple- it loads up python and imports the module instead of dumping the code into __main__. In practice that has many effects. It’s a different import system that behaves differently. Some of those changes were not intentional and were only discovered after implementation. Python’s imports are a mess and it’s okay to be confused.
Amazing this is so hard to answer when some people say the question is too simple to bother with.
As far as I can tell, the practical purpose is so you can use dot notation to run your script when you aren’t in the directory.
You can run
python -m path.to.my.happy.place
instead of being in path/to/my/happy and running python place.py
When I use python -m pip install <package>, how is that different from using just pip install <package>? Similarly, why would I write python -m pip install --upgrade pip to upgrade Pip, instead of just pip install --upgrade pip?
If you type python --help
You get
// More flags above
-m mod : run library module as a script (terminates option list)
// and more flags below
A great many things in a terminal will show you how to use it if you either use command --help or man command
The -m stands for module-name.
From Command line and environment:
python [-bBdEhiIOqsSuvVWx?] [-c command | -m module-name | script | – ] [args]
From Python Docs:
Since the argument is a module name, you must not give a file extension (.py). The
module-nameshould be a valid Python module name, but the implementation may not always enforce this (e.g. it may allow you to use a name that includes a hyphen).Package names are also permitted. When a package name is supplied instead of a normal module, the interpreter will execute
<pkg>.__main__as the main module. This behaviour is deliberately similar to the handling of directories and zipfiles that are passed to the interpreter as the script argument.
When -m is used with a python statement on the command line, followed by a <module_name>, then it enables the module to be executed as an executable file.
You can refer to python docs for the same, or run python --help
If you have multiple versions of python installed and you want to upgrade pip pip install --upgrade pip how do you know which python version will be affected? it depends on path variable for the shell. You might also get warning in this case. To avoid this confusion use -m then it looks in variable sys.path. This is another advantage of -m.
# importing module
import sys
# printing all directories for
# interpreter to search
sys.path
Consider the following scenario.
You have three versions of Python installed:
- Python 3.7
- Python 3.8
- Python 3.9
Your "default" version is 3.8. It’s the first one appearing in your path. Therefore, when you type python3 (Linux or Mac) or python (Windows) in a shell you will start a 3.8 interpreter because that’s the first Python executable that is found when traversing your path.
Suppose you are then starting a new project where you want to use Python 3.9. You create a virtual environment called .venv and activate it.
python3.9 -m venv .venv # "py -3.9" on Windows
source .venv/bin/activate # ".venvScriptsactivate" on Windows
We now have the virtual environment activated using Python 3.9. Typing python in a shell starts the 3.9 interpreter.
BUT, if you type
pip install <some-package>
Then what version of pip is used? Is it the pip for the default version, i.e. Python 3.8, or the Python version within the virtual environment?
An easy way to get around that ambiguity is simply to use
python -m pip install <some-package>
The -m flag makes sure that you are using the pip that’s tied to the active Python executable.
It’s good practice to always use -m, even if you have just one global version of Python installed from which you create virtual environments.
Re. path
The so-called path is a list of directories where your system searches for executables. When you type a command, like python, this list is traversed from the first directory to the last, searching for a filename that matches the command you typed.
If the filename/command is found, the matched file gets executed without taking into account potential later matches. If no match occurs, you get a Command not found or a variation thereof. This behavior is by design.
On UNIX systems the path environment variable is called $PATH, while on Windows systems it’s referred to as %PATH%
More general comments about the -m-flag (Dec. 2022)
Most people viewing this will likely want the explanation given above with pip. In a more general sense though, when using python -m some_module, the -m flag makes Python execute some_module as a script. This is stated in the docs, but might be difficult to understand without some baseline knowledge.
What does it mean to "run as a script"?
In Python, a module some_module is typically imported into another Python file with an import some_module statement at the top of the importing file. This enables the use of functions, classes, and variables defined in some_module inside the importing file.
To execute some_module as a script instead of importing it, you would define an if __name__ == "__main__" block inside the file. This block gets executed when running python some_module.py on the command line. This is useful because you do not want this code block to run when importing into other files, but you do want it to run when invoked from the command line.
For modules inside your project, this script/module construct should run as is, because Python will find the module from your working directory when running from the terminal:
python some_module.py
But for modules that are part of Python’s Standard Library, this will not work. The example from the Python docs uses timeit (pip works the same):
python3 timeit -s 'print("hello")' # 'python timeit.py ...' fails as well
This returns the error: "python: can't open file '/home/<username>/timeit': [Errno 2] No such file or directory"
Adding the -m flag tells Python to look in path for timeit.py and execute the if __name__ == "__main__" clause from the file.
python3 -m timeit -s 'print("hello")'
This works as expected.
The source for the if __name__ == "__main__" block for the timeit module can be found here.
This is actually an interesting question, so let’s explore pep 338 linked by @jedwards in the top comment.
The -m flag originally served a simpler purpose- to convert a module name into a script name. In python 2.4, the behavior was:
the command line is effectively reinterpreted from python <options> -m
<module> <args> to python <options> <filename> <args>.
This doesn’t seem very useful, but that’s what it did back then. Pep 338 extends this behavior further.
The semantics proposed are fairly simple [sic]: if -m is used to
execute a module the PEP 302 import mechanisms are used to locate the
module and retrieve its compiled code, before executing the module in
accordance with the semantics for a top-level module.
It goes on to explain further that python will identify the package the module is from, import that package using the standard process, and run the module. From what I understand calling "python3 -m package.module" is the same as calling:
python3
from package import module
The -m flag will run the module as __file__ and not __main__. It will also insert the local directory into sys.path instead of the script’s directory. It therefore breaks relative imports, although this was not intentional, and the authors therefore recommend to always use absolute imports. It also depends on how you call it- "python3 -m package.module" is not the same as "python3 -m module".
In theory it’s simple- it loads up python and imports the module instead of dumping the code into __main__. In practice that has many effects. It’s a different import system that behaves differently. Some of those changes were not intentional and were only discovered after implementation. Python’s imports are a mess and it’s okay to be confused.
Amazing this is so hard to answer when some people say the question is too simple to bother with.
As far as I can tell, the practical purpose is so you can use dot notation to run your script when you aren’t in the directory.
You can run
python -m path.to.my.happy.place
instead of being in path/to/my/happy and running python place.py