How to debug a Python module run with python -m from the command line?
Question:
I know that a Python script can be debugged from the command line with
python -m pdb my_script.py
if my_script.py is a script intended to be run with python my_script.py.
However, a python module my_module.py should be run with python -m my_module. Even scripts that contain relative imports should be run with python -m. How can I run python -m my_module under pdb‘s control? The following does not work:
python -m pdb -m my_module
Answers:
You can add pdb.set_trace() in your code for interactive debugging, before the code you want to debug.
class C:
def __init__(self, x):
self.x = x
def inst_f(self):
pass
a = C('this is a')
import pdb
pdb.set_trace()
b = C('this is b')
print a.x is b.x
Running this will output
> c:python27teststest.py(11)<module>()
-> b = C('this is b')
(Pdb)
And let you use python debugger.
According to the python command-line manpage the -m flag does the following:
Searches sys.path for the named module and runs the corresponding .py file as a script.
Given this, I would feel confident debugging by running the .py file as per your first example. One thing to keep in mind is that -m searches sys.path. Luckily, python looks at the current working directory first, so as long as the .py you are debugging is in your cwd, python -m module and python module.py equivalent.
You can’t do it now, because -m terminates option list
python -h
...
-m mod : run library module as a script (terminates option list)
...
That means it’s mod’s job to interpret the rest of the arguments list and this behavior fully depends on how mod is designed internally and whether it support another -m
Lets check out what’s happening inside pdb of python 2.x. Actually, nothing intereseting, it only expects a script name to be supplied:
if not sys.argv[1:] or sys.argv[1] in ("--help", "-h"):
print "usage: pdb.py scriptfile [arg] ..."
sys.exit(2)
mainpyfile = sys.argv[1] # Get script filename
if not os.path.exists(mainpyfile):
print 'Error:', mainpyfile, 'does not exist'
sys.exit(1)
del sys.argv[0] # Hide "pdb.py" from argument list
# Replace pdb's dir with script's dir in front of module search path.
sys.path[0] = os.path.dirname(mainpyfile)
# Note on saving/restoring sys.argv: it's a good idea when sys.argv was
# modified by the script being debugged. It's a bad idea when it was
# changed by the user from the command line. There is a "restart" command
# which allows explicit specification of command line arguments.
pdb = Pdb()
while True:
try:
pdb._runscript(mainpyfile)
Same for the currently released versions of python 3.x
Good news
The pull request that allows to do what you’re asking has been merged 5 days ago. What a mysterious coincidence! Here’s the code
So just wait a bit for the upcoming python 3.x versions to have this issue resolved )
The following script will run a module and break into post-mortem debugging if an exception occurs while running the module. It should work both with Python 2.7 and 3.x.
Usage:
mdb.py module_name [args ...]
Known limitations:
- While running the module code,
sys.argv[0] is preserved as the module name, instead of being resolved to the file path of the module.
- If the target module is not found, the error is not reported any differently than if the error occurred during the execution of the module
mdb.py
#!/usr/bin/env python
from __future__ import print_function
import pdb
import runpy
import sys
import traceback
if len(sys.argv) == 0:
print("Usage: mdb.py module_name [args ...]")
exit(1)
modulename = sys.argv[1]
del sys.argv[0]
try:
runpy.run_module(modulename, run_name='__main__')
except:
traceback.print_exception(*sys.exc_info())
print("")
print("-" * 40)
print("mdb: An exception occurred while executing module ", modulename)
print("mdb: See the traceback above.")
print("mdb: Entering post-mortem debugging.")
print("-" * 40)
pdb.post_mortem(sys.exc_info()[2])
Demonstration:
$ tree
.
├── mdb.py
└── mypackage
├── __init__.py
├── __main__.py
└── mymodule.py
1 directory, 4 files
$ ###################### Examine the module code ###################
$ cat mypackage/mymodule.py
from __future__ import print_function
import sys
print("mymodule loaded")
if __name__ == "__main__":
print("mymodule executed")
print("args:", sys.argv)
$ #################### Run the module through python ###############
$ python -m mypackage.mymodule abc defgh
mymodule loaded
mymodule executed
args: ['/home/leon/playground/mdb/mypackage/mymodule.py', 'abc', 'defgh']
$ #################### Run the module through mdb ##################
$ ./mdb.py mypackage.mymodule abc defgh
mymodule loaded
mymodule executed
args: ['mypackage.mymodule', 'abc', 'defgh']
$ ### ^^^^^^^^^^^^^^^^^^
$ ### Note that sys.argv[0] is not resolved to the file path
$ ###################### Examine the module code ###################
$ cat mypackage/__main__.py
from __future__ import print_function
import sys
print("mypackage loaded")
if __name__ == "__main__":
print("mypackage executed")
print("args:", sys.argv)
print(x + y)
$ #################### Run the module through python ###############
$ python -m mypackage
mypackage loaded
mypackage executed
args: ['/home/leon/playground/mdb/mypackage/__main__.py']
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/leon/playground/mdb/mypackage/__main__.py", line 9, in <module>
print(x + y)
NameError: name 'x' is not defined
$ #################### Run the module through mdb ##################
$ ./mdb.py mypackage
mypackage loaded
mypackage executed
args: ['mypackage']
Traceback (most recent call last):
File "./mdb.py", line 17, in <module>
runpy.run_module(modulename, run_name='__main__')
File "/usr/lib/python2.7/runpy.py", line 192, in run_module
fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/leon/playground/mdb/mypackage/__main__.py", line 9, in <module>
print(x + y)
NameError: name 'x' is not defined
----------------------------------------
mdb: An exception occurred while executing module mypackage
mdb: See the traceback above.
mdb: Entering post-mortem debugging.
----------------------------------------
> /home/leon/playground/mdb/mypackage/__main__.py(9)<module>()
-> print(x + y)
(Pdb) q
Python 3.7 adds that feature
From the docs, it looks that your command:
python -m pdb -m my_module
will start working on Python 3.7:
New in version 3.7: pdb.py now accepts a -m option that execute modules similar to the way python3 -m does. As with a script, the debugger will pause execution just before the first line of the module.
As others have said, this feature is added in Python 3.7. However, people are still using older versions, so hopefully the solution below helps those people!
Basically you can create a script that runs the module just by importing the __main__.py file for the package:
# runner.py
import path.to.my.module.__main__
Then simply run that script under pdb: python -m pdb runner.py [args].
This solution works with relative imports (because __main__ is running under the package path rather than being called as a script).
The reason you just need to import the __main__ module is that Python imports trigger the top-level code to be executed, and in __main__ the logic is performed at the top-level.
I know that a Python script can be debugged from the command line with
python -m pdb my_script.py
if my_script.py is a script intended to be run with python my_script.py.
However, a python module my_module.py should be run with python -m my_module. Even scripts that contain relative imports should be run with python -m. How can I run python -m my_module under pdb‘s control? The following does not work:
python -m pdb -m my_module
You can add pdb.set_trace() in your code for interactive debugging, before the code you want to debug.
class C:
def __init__(self, x):
self.x = x
def inst_f(self):
pass
a = C('this is a')
import pdb
pdb.set_trace()
b = C('this is b')
print a.x is b.x
Running this will output
> c:python27teststest.py(11)<module>()
-> b = C('this is b')
(Pdb)
And let you use python debugger.
According to the python command-line manpage the -m flag does the following:
Searches sys.path for the named module and runs the corresponding .py file as a script.
Given this, I would feel confident debugging by running the .py file as per your first example. One thing to keep in mind is that -m searches sys.path. Luckily, python looks at the current working directory first, so as long as the .py you are debugging is in your cwd, python -m module and python module.py equivalent.
You can’t do it now, because -m terminates option list
python -h
...
-m mod : run library module as a script (terminates option list)
...
That means it’s mod’s job to interpret the rest of the arguments list and this behavior fully depends on how mod is designed internally and whether it support another -m
Lets check out what’s happening inside pdb of python 2.x. Actually, nothing intereseting, it only expects a script name to be supplied:
if not sys.argv[1:] or sys.argv[1] in ("--help", "-h"):
print "usage: pdb.py scriptfile [arg] ..."
sys.exit(2)
mainpyfile = sys.argv[1] # Get script filename
if not os.path.exists(mainpyfile):
print 'Error:', mainpyfile, 'does not exist'
sys.exit(1)
del sys.argv[0] # Hide "pdb.py" from argument list
# Replace pdb's dir with script's dir in front of module search path.
sys.path[0] = os.path.dirname(mainpyfile)
# Note on saving/restoring sys.argv: it's a good idea when sys.argv was
# modified by the script being debugged. It's a bad idea when it was
# changed by the user from the command line. There is a "restart" command
# which allows explicit specification of command line arguments.
pdb = Pdb()
while True:
try:
pdb._runscript(mainpyfile)
Same for the currently released versions of python 3.x
Good news
The pull request that allows to do what you’re asking has been merged 5 days ago. What a mysterious coincidence! Here’s the code
So just wait a bit for the upcoming python 3.x versions to have this issue resolved )
The following script will run a module and break into post-mortem debugging if an exception occurs while running the module. It should work both with Python 2.7 and 3.x.
Usage:
mdb.py module_name [args ...]
Known limitations:
- While running the module code,
sys.argv[0]is preserved as the module name, instead of being resolved to the file path of the module. - If the target module is not found, the error is not reported any differently than if the error occurred during the execution of the module
mdb.py
#!/usr/bin/env python
from __future__ import print_function
import pdb
import runpy
import sys
import traceback
if len(sys.argv) == 0:
print("Usage: mdb.py module_name [args ...]")
exit(1)
modulename = sys.argv[1]
del sys.argv[0]
try:
runpy.run_module(modulename, run_name='__main__')
except:
traceback.print_exception(*sys.exc_info())
print("")
print("-" * 40)
print("mdb: An exception occurred while executing module ", modulename)
print("mdb: See the traceback above.")
print("mdb: Entering post-mortem debugging.")
print("-" * 40)
pdb.post_mortem(sys.exc_info()[2])
Demonstration:
$ tree
.
├── mdb.py
└── mypackage
├── __init__.py
├── __main__.py
└── mymodule.py
1 directory, 4 files
$ ###################### Examine the module code ###################
$ cat mypackage/mymodule.py
from __future__ import print_function
import sys
print("mymodule loaded")
if __name__ == "__main__":
print("mymodule executed")
print("args:", sys.argv)
$ #################### Run the module through python ###############
$ python -m mypackage.mymodule abc defgh
mymodule loaded
mymodule executed
args: ['/home/leon/playground/mdb/mypackage/mymodule.py', 'abc', 'defgh']
$ #################### Run the module through mdb ##################
$ ./mdb.py mypackage.mymodule abc defgh
mymodule loaded
mymodule executed
args: ['mypackage.mymodule', 'abc', 'defgh']
$ ### ^^^^^^^^^^^^^^^^^^
$ ### Note that sys.argv[0] is not resolved to the file path
$ ###################### Examine the module code ###################
$ cat mypackage/__main__.py
from __future__ import print_function
import sys
print("mypackage loaded")
if __name__ == "__main__":
print("mypackage executed")
print("args:", sys.argv)
print(x + y)
$ #################### Run the module through python ###############
$ python -m mypackage
mypackage loaded
mypackage executed
args: ['/home/leon/playground/mdb/mypackage/__main__.py']
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/leon/playground/mdb/mypackage/__main__.py", line 9, in <module>
print(x + y)
NameError: name 'x' is not defined
$ #################### Run the module through mdb ##################
$ ./mdb.py mypackage
mypackage loaded
mypackage executed
args: ['mypackage']
Traceback (most recent call last):
File "./mdb.py", line 17, in <module>
runpy.run_module(modulename, run_name='__main__')
File "/usr/lib/python2.7/runpy.py", line 192, in run_module
fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/leon/playground/mdb/mypackage/__main__.py", line 9, in <module>
print(x + y)
NameError: name 'x' is not defined
----------------------------------------
mdb: An exception occurred while executing module mypackage
mdb: See the traceback above.
mdb: Entering post-mortem debugging.
----------------------------------------
> /home/leon/playground/mdb/mypackage/__main__.py(9)<module>()
-> print(x + y)
(Pdb) q
Python 3.7 adds that feature
From the docs, it looks that your command:
python -m pdb -m my_module
will start working on Python 3.7:
New in version 3.7: pdb.py now accepts a -m option that execute modules similar to the way python3 -m does. As with a script, the debugger will pause execution just before the first line of the module.
As others have said, this feature is added in Python 3.7. However, people are still using older versions, so hopefully the solution below helps those people!
Basically you can create a script that runs the module just by importing the __main__.py file for the package:
# runner.py
import path.to.my.module.__main__
Then simply run that script under pdb: python -m pdb runner.py [args].
This solution works with relative imports (because __main__ is running under the package path rather than being called as a script).
The reason you just need to import the __main__ module is that Python imports trigger the top-level code to be executed, and in __main__ the logic is performed at the top-level.