How do I profile a Python script?
Question:
Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With Python, sometimes the approaches are somewhat kludgey – i.e., adding timing code to __main__
.
What is a good way to profile how long a Python program takes to run?
Answers:
Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.
You can call it from within your code, or from the interpreter, like this:
import cProfile
cProfile.run('foo()')
Even more usefully, you can invoke the cProfile when running a script:
python -m cProfile myscript.py
To make it even easier, I made a little batch file called ‘profile.bat’:
python -m cProfile %1
So all I have to do is run:
profile euler048.py
And I get this:
1007 function calls in 0.061 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.061 0.061 <string>:1(<module>)
1000 0.051 0.000 0.051 0.000 euler048.py:2(<lambda>)
1 0.005 0.005 0.061 0.061 euler048.py:2(<module>)
1 0.000 0.000 0.061 0.061 {execfile}
1 0.002 0.002 0.053 0.053 {map}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler objects}
1 0.000 0.000 0.000 0.000 {range}
1 0.003 0.003 0.003 0.003 {sum}
EDIT: Updated link to a good video resource from PyCon 2013 titled
Python Profiling
Also via YouTube.
In Virtaal’s source there’s a very useful class and decorator that can make profiling (even for specific methods/functions) very easy. The output can then be viewed very comfortably in KCacheGrind.
It’s worth pointing out that using the profiler only works (by default) on the main thread, and you won’t get any information from other threads if you use them. This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation.
If you also want to profile threads, you’ll want to look at the threading.setprofile()
function in the docs.
You could also create your own threading.Thread
subclass to do it:
class ProfiledThread(threading.Thread):
# Overrides threading.Thread.run()
def run(self):
profiler = cProfile.Profile()
try:
return profiler.runcall(threading.Thread.run, self)
finally:
profiler.dump_stats('myprofile-%d.profile' % (self.ident,))
and use that ProfiledThread
class instead of the standard one. It might give you more flexibility, but I’m not sure it’s worth it, especially if you are using third-party code which wouldn’t use your class.
The python wiki is a great page for profiling resources:
http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code
as is the python docs:
http://docs.python.org/library/profile.html
as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen:
python -m cProfile -s time mine.py <args>
or to file:
python -m cProfile -o output.file mine.py <args>
PS> If you are using Ubuntu, make sure to install python-profile
apt-get install python-profiler
If you output to file you can get nice visualizations using the following tools
PyCallGraph : a tool to create call graph images
install:
pip install pycallgraph
run:
pycallgraph mine.py args
view:
gimp pycallgraph.png
You can use whatever you like to view the png file, I used gimp
Unfortunately I often get
dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.257079 to fit
which makes my images unusably small. So I generally create svg files:
pycallgraph -f svg -o pycallgraph.svg mine.py <args>
PS> make sure to install graphviz (which provides the dot program):
pip install graphviz
Alternative Graphing using gprof2dot via @maxy / @quodlibetor :
pip install gprof2dot
python -m cProfile -o profile.pstats mine.py
gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg
A nice profiling module is the line_profiler (called using the script kernprof.py). It can be downloaded here.
My understanding is that cProfile only gives information about total time spent in each function. So individual lines of code are not timed. This is an issue in scientific computing since often one single line can take a lot of time. Also, as I remember, cProfile didn’t catch the time I was spending in say numpy.dot.
Following Joe Shaw’s answer about multi-threaded code not to work as expected, I figured that the runcall
method in cProfile is merely doing self.enable()
and self.disable()
calls around the profiled function call, so you can simply do that yourself and have whatever code you want in-between with minimal interference with existing code.
A while ago I made pycallgraph
which generates a visualisation from your Python code. Edit: I’ve updated the example to work with 3.3, the latest release as of this writing.
After a pip install pycallgraph
and installing GraphViz you can run it from the command line:
pycallgraph graphviz -- ./mypythonscript.py
Or, you can profile particular parts of your code:
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
with PyCallGraph(output=GraphvizOutput()):
code_to_profile()
Either of these will generate a pycallgraph.png
file similar to the image below:

Ever want to know what the hell that python script is doing? Enter the
Inspect Shell. Inspect Shell lets you print/alter globals and run
functions without interrupting the running script. Now with
auto-complete and command history (only on linux).
Inspect Shell is not a pdb-style debugger.
https://github.com/amoffat/Inspect-Shell
You could use that (and your wristwatch).
@Maxy’s comment on this answer helped me out enough that I think it deserves its own answer: I already had cProfile-generated .pstats files and I didn’t want to re-run things with pycallgraph, so I used gprof2dot, and got pretty svgs:
$ sudo apt-get install graphviz
$ git clone https://github.com/jrfonseca/gprof2dot
$ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin
$ cd $PROJECT_DIR
$ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg
and BLAM!
It uses dot (the same thing that pycallgraph uses) so output looks similar. I get the impression that gprof2dot loses less information though:
My way is to use yappi (https://github.com/sumerc/yappi). It’s especially useful combined with an RPC server where (even just for debugging) you register method to start, stop and print profiling information, e.g. in this way:
@staticmethod
def startProfiler():
yappi.start()
@staticmethod
def stopProfiler():
yappi.stop()
@staticmethod
def printProfiler():
stats = yappi.get_stats(yappi.SORTTYPE_TTOT, yappi.SORTORDER_DESC, 20)
statPrint = 'n'
namesArr = [len(str(stat[0])) for stat in stats.func_stats]
log.debug("namesArr %s", str(namesArr))
maxNameLen = max(namesArr)
log.debug("maxNameLen: %s", maxNameLen)
for stat in stats.func_stats:
nameAppendSpaces = [' ' for i in range(maxNameLen - len(stat[0]))]
log.debug('nameAppendSpaces: %s', nameAppendSpaces)
blankSpace = ''
for space in nameAppendSpaces:
blankSpace += space
log.debug("adding spaces: %s", len(nameAppendSpaces))
statPrint = statPrint + str(stat[0]) + blankSpace + " " + str(stat[1]).ljust(8) + "t" + str(
round(stat[2], 2)).ljust(8 - len(str(stat[2]))) + "t" + str(round(stat[3], 2)) + "n"
log.log(1000, "nname" + ''.ljust(maxNameLen - 4) + " ncall tttot ttsub")
log.log(1000, statPrint)
Then when your program work you can start profiler at any time by calling the startProfiler
RPC method and dump profiling information to a log file by calling printProfiler
(or modify the rpc method to return it to the caller) and get such output:
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
name ncall ttot tsub
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
C:Python27libsched.py.run:80 22 0.11 0.05
M: 2_documents_repos 9_aheadReposappsahdModbusSrvpyAheadRpcSrvxmlRpc.py.iterFnc:293 22 0.11 0.0
M: 2_documents_repos 9_aheadReposappsahdModbusSrvserverMain.py.makeIteration:515 22 0.11 0.0
M: 2_documents_repos 9_aheadReposappsahdModbusSrvpyAheadRpcSrvPicklingXMLRPC.py._dispatch:66 1 0.0 0.0
C:Python27libBaseHTTPServer.py.date_time_string:464 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil_psmswindows.py._get_raw_meminfo:243 4 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.decode_request_content:537 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil_psmswindows.py.get_system_cpu_times:148 4 0.0 0.0
<string>.__new__:8 220 0.0 0.0
C:Python27libsocket.py.close:276 4 0.0 0.0
C:Python27libthreading.py.__init__:558 1 0.0 0.0
<string>.__new__:8 4 0.0 0.0
C:Python27libthreading.py.notify:372 1 0.0 0.0
C:Python27librfc822.py.getheader:285 4 0.0 0.0
C:Python27libBaseHTTPServer.py.handle_one_request:301 1 0.0 0.0
C:Python27libxmlrpclib.py.end:816 3 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.do_POST:467 1 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.is_rpc_path_valid:460 1 0.0 0.0
C:Python27libSocketServer.py.close_request:475 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil__init__.py.cpu_times:1066 4 0.0 0.0
It may not be very useful for short scripts but helps to optimize server-type processes especially given the printProfiler
method can be called multiple times over time to profile and compare e.g. different program usage scenarios.
In newer versions of yappi, the following code will work:
@staticmethod
def printProfile():
yappi.get_func_stats().print_all()
Also worth mentioning is the GUI cProfile dump viewer RunSnakeRun. It allows you to sort and select, thereby zooming in on the relevant parts of the program. The sizes of the rectangles in the picture is proportional to the time taken. If you mouse over a rectangle it highlights that call in the table and everywhere on the map. When you double-click on a rectangle it zooms in on that portion. It will show you who calls that portion and what that portion calls.
The descriptive information is very helpful. It shows you the code for that bit which can be helpful when you are dealing with built-in library calls. It tells you what file and what line to find the code.
Also want to point at that the OP said ‘profiling’ but it appears he meant ‘timing’. Keep in mind programs will run slower when profiled.

pprofile
line_profiler
(already presented here) also inspired pprofile
, which is described as:
Line-granularity, thread-aware deterministic and statistic pure-python
profiler
It provides line-granularity as line_profiler
, is pure Python, can be used as a standalone command or a module, and can even generate callgrind-format files that can be easily analyzed with [k|q]cachegrind
.
vprof
There is also vprof, a Python package described as:
[…] providing rich and interactive visualizations for various Python program characteristics such as running time and memory usage.
To add on to https://stackoverflow.com/a/582337/1070617,
I wrote this module that allows you to use cProfile and view its output easily. More here: https://github.com/ymichael/cprofilev
$ python -m cprofilev /your/python/program
# Go to http://localhost:4000 to view collected statistics.
Also see: http://ymichael.com/2014/03/08/profiling-python-with-cprofile.html on how to make sense of the collected statistics.
cProfile is great for quick profiling but most of the time it was ending for me with the errors. Function runctx solves this problem by initializing correctly the environment and variables, hope it can be useful for someone:
import cProfile
cProfile.runctx('foo()', None, locals())
A new tool to handle profiling in Python is PyVmMonitor: http://www.pyvmmonitor.com/
It has some unique features such as
- Attach profiler to a running (CPython) program
- On demand profiling with Yappi integration
- Profile on a different machine
- Multiple processes support (multiprocessing, django…)
- Live sampling/CPU view (with time range selection)
- Deterministic profiling through cProfile/profile integration
- Analyze existing PStats results
- Open DOT files
- Programatic API access
- Group samples by method or line
- PyDev integration
- PyCharm integration
Note: it’s commercial, but free for open source.
There’s a lot of great answers but they either use command line or some external program for profiling and/or sorting the results.
I really missed some way I could use in my IDE (eclipse-PyDev) without touching the command line or installing anything. So here it is.
Profiling without command line
def count():
from math import sqrt
for x in range(10**5):
sqrt(x)
if __name__ == '__main__':
import cProfile, pstats
cProfile.run("count()", "{}.profile".format(__file__))
s = pstats.Stats("{}.profile".format(__file__))
s.strip_dirs()
s.sort_stats("time").print_stats(10)
See docs or other answers for more info.
There’s also a statistical profiler called statprof
. It’s a sampling profiler, so it adds minimal overhead to your code and gives line-based (not just function-based) timings. It’s more suited to soft real-time applications like games, but may be have less precision than cProfile.
The version in pypi is a bit old, so can install it with pip
by specifying the git repository:
pip install git+git://github.com/bos/[email protected]
You can run it like this:
import statprof
with statprof.profile():
my_questionable_function()
cProfile
is great for profiling, while kcachegrind
is great for visualizing the results. The pyprof2calltree
in between handles the file conversion.
python -m cProfile -o script.profile script.py
pyprof2calltree -i script.profile -o script.calltree
kcachegrind script.calltree
Required system packages:
kcachegrind
(Linux), qcachegrind
(MacOs)
Setup on Ubuntu:
apt-get install kcachegrind
pip install pyprof2calltree
The result:
I ran into a handy tool called SnakeViz when researching this topic. SnakeViz is a web-based profiling visualization tool. It is very easy to install and use. The usual way I use it is to generate a stat file with %prun
and then do analysis in SnakeViz.
The main viz technique used is Sunburst chart as shown below, in which the hierarchy of function calls is arranged as layers of arcs and time info encoded in their angular widths.
The best thing is you can interact with the chart. For example, to zoom in one can click on an arc, and the arc and its descendants will be enlarged as a new sunburst to display more details.
When i’m not root on the server, I use
lsprofcalltree.py and run my program like this:
python lsprofcalltree.py -o callgrind.1 test.py
Then I can open the report with any callgrind-compatible software, like qcachegrind
It would depend on what you want to see out of profiling. Simple time
metrics can be given by (bash).
time python python_prog.py
Even ‘/usr/bin/time’ can output detailed metrics by using ‘–verbose’ flag.
To check time metrics given by each function and to better understand how much time is spent on functions, you can use the inbuilt cProfile in python.
Going into more detailed metrics like performance, time is not the only metric. You can worry about memory, threads etc.
Profiling options:
1. line_profiler is another profiler used commonly to find out timing metrics line-by-line.
2. memory_profiler is a tool to profile memory usage.
3. heapy (from project Guppy) Profile how objects in the heap are used.
These are some of the common ones I tend to use. But if you want to find out more, try reading this book
It is a pretty good book on starting out with performance in mind. You can move onto advanced topics on using Cython and JIT(Just-in-time) compiled python.
Simplest and quickest way to find where all the time is going.
1. pip install snakeviz
2. python -m cProfile -o temp.dat <PROGRAM>.py
3. snakeviz temp.dat
Draws a pie chart in a browser. Biggest piece is the problem function. Very simple.
I recently created tuna for visualizing Python runtime and import profiles; this may be helpful here.
Install with
pip install tuna
Create a runtime profile
python3 -m cProfile -o program.prof yourfile.py
or an import profile (Python 3.7+ required)
python3 -X importprofile yourfile.py 2> import.log
Then just run tuna on the file
tuna program.prof
gprof2dot_magic
Magic function for gprof2dot
to profile any Python statement as a DOT graph in JupyterLab or Jupyter Notebook.
GitHub repo: https://github.com/mattijn/gprof2dot_magic
installation
Make sure you’ve the Python package gprof2dot_magic
.
pip install gprof2dot_magic
Its dependencies gprof2dot
and graphviz
will be installed as well
usage
To enable the magic function, first load the gprof2dot_magic
module
%load_ext gprof2dot_magic
and then profile any line statement as a DOT graph as such:
%gprof2dot print('hello world')
The terminal-only (and simplest) solution, in case all those fancy UI’s fail to install or to run:
ignore cProfile
completely and replace it with pyinstrument
, that will collect and display the tree of calls right after execution.
Install:
$ pip install pyinstrument
Profile and display result:
$ python -m pyinstrument ./prog.py
Works with python2 and 3.
[EDIT]
The documentation of the API, for profiling only a part of the code, can be found here.
If you want to make a cumulative profiler,
meaning to run the function several times in a row and watch the sum of the results.
you can use this cumulative_profiler
decorator:
it’s python >= 3.6 specific, but you can remove nonlocal
for it work on older versions.
import cProfile, pstats
class _ProfileFunc:
def __init__(self, func, sort_stats_by):
self.func = func
self.profile_runs = []
self.sort_stats_by = sort_stats_by
def __call__(self, *args, **kwargs):
pr = cProfile.Profile()
pr.enable() # this is the profiling section
retval = self.func(*args, **kwargs)
pr.disable()
self.profile_runs.append(pr)
ps = pstats.Stats(*self.profile_runs).sort_stats(self.sort_stats_by)
return retval, ps
def cumulative_profiler(amount_of_times, sort_stats_by='time'):
def real_decorator(function):
def wrapper(*args, **kwargs):
nonlocal function, amount_of_times, sort_stats_by # for python 2.x remove this row
profiled_func = _ProfileFunc(function, sort_stats_by)
for i in range(amount_of_times):
retval, ps = profiled_func(*args, **kwargs)
ps.print_stats()
return retval # returns the results of the function
return wrapper
if callable(amount_of_times): # incase you don't want to specify the amount of times
func = amount_of_times # amount_of_times is the function in here
amount_of_times = 5 # the default amount
return real_decorator(func)
return real_decorator
Example
profiling the function baz
import time
@cumulative_profiler
def baz():
time.sleep(1)
time.sleep(2)
return 1
baz()
baz
ran 5 times and printed this:
20 function calls in 15.003 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
10 15.003 1.500 15.003 1.500 {built-in method time.sleep}
5 0.000 0.000 15.003 3.001 <ipython-input-9-c89afe010372>:3(baz)
5 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
specifying the amount of times
@cumulative_profiler(3)
def baz():
...
I just developed my own profiler inspired from pypref_time:
https://github.com/modaresimr/auto_profiler
Update Version 2
Install:
pip install auto_profiler
Quick Start:
from auto_profiler import Profiler
with Profiler():
your_function()
Using in Jupyter, let you have realtime view of elapsed times
Update Version 1
By adding a decorator it will show a tree of time-consuming functions
@Profiler(depth=4)
Install by: pip install auto_profiler
Example
import time # line number 1
import random
from auto_profiler import Profiler, Tree
def f1():
mysleep(.6+random.random())
def mysleep(t):
time.sleep(t)
def fact(i):
f1()
if(i==1):
return 1
return i*fact(i-1)
def main():
for i in range(5):
f1()
fact(3)
with Profiler(depth=4):
main()
Example Output
Time [Hits * PerHit] Function name [Called from] [function location]
-----------------------------------------------------------------------
8.974s [1 * 8.974] main [auto-profiler/profiler.py:267] [/test/t2.py:30]
├── 5.954s [5 * 1.191] f1 [/test/t2.py:34] [/test/t2.py:14]
│ └── 5.954s [5 * 1.191] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 5.954s [5 * 1.191] <time.sleep>
|
|
| # The rest is for the example recursive function call fact
└── 3.020s [1 * 3.020] fact [/test/t2.py:36] [/test/t2.py:20]
├── 0.849s [1 * 0.849] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 0.849s [1 * 0.849] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 0.849s [1 * 0.849] <time.sleep>
└── 2.171s [1 * 2.171] fact [/test/t2.py:24] [/test/t2.py:20]
├── 1.552s [1 * 1.552] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 1.552s [1 * 1.552] mysleep [/test/t2.py:15] [/test/t2.py:17]
└── 0.619s [1 * 0.619] fact [/test/t2.py:24] [/test/t2.py:20]
└── 0.619s [1 * 0.619] f1 [/test/t2.py:21] [/test/t2.py:14]
With a statistical profiler like austin, no instrumentation is required, meaning that you can get profiling data out of a Python application simply with
austin python3 my_script.py
The raw output isn’t very useful, but you can pipe that to flamegraph.pl
to get a flame graph representation of that data that gives you a breakdown of where the time (measured in microseconds of real time) is being spent.
austin python3 my_script.py | flamegraph.pl > my_script_profile.svg
Alternatively, you can also use the web application Speedscope.app for quick visualisation of the collected samples. If you have pprof installed, you can also get austin-python (with e.g. pipx install austin-python
) and use the austin2pprof
to covert to the pprof format.
However, if you have VS Code installed you could use the Austin extension for a more interactive experience, with source code heat maps, top functions and collected call stacks

If you’d rather use the terminal, you can also use the TUI, that also has a live graph mode:

For getting quick profile stats on an IPython notebook.
One can embed line_profiler and memory_profiler straight into their notebooks.
Another useful package is Pympler. It is a powerful profiling package that’s capable to track classes,objects,functions,memory leaks etc. Examples below, Docs attached.
Get it!
!pip install line_profiler
!pip install memory_profiler
!pip install pympler
Load it!
%load_ext line_profiler
%load_ext memory_profiler
Use it!
%time
%time print('Outputs CPU time,Wall Clock time')
#CPU times: user 2 µs, sys: 0 ns, total: 2 µs Wall time: 5.96 µs
Gives:
- CPU times: CPU level execution time
- sys times: system level execution time
- total: CPU time + system time
- Wall time: Wall Clock Time
%timeit
%timeit -r 7 -n 1000 print('Outputs execution time of the snippet')
#1000 loops, best of 7: 7.46 ns per loop
- Gives best time out of given number of runs(r) in looping (n) times.
- Outputs details on system caching:
- When code snippets are executed multiple times, system caches a few opearations and doesn’t execute them again that may hamper the accuracy of the profile reports.
%prun
%prun -s cumulative 'Code to profile'
Gives:
- number of function calls(ncalls)
- has entries per function call(distinct)
- time taken per call(percall)
- time elapsed till that function call(cumtime)
- name of the func/module called etc…
%memit
%memit 'Code to profile'
#peak memory: 199.45 MiB, increment: 0.00 MiB
Gives:
- Memory usage
%lprun
#Example function
def fun():
for i in range(10):
print(i)
#Usage: %lprun <name_of_the_function> function
%lprun -f fun fun()
Gives:
- Line wise stats
sys.getsizeof
sys.getsizeof('code to profile')
# 64 bytes
Returns the size of an object in bytes.
asizeof() from pympler
from pympler import asizeof
obj = [1,2,("hey","ha"),3]
print(asizeof.asizeof(obj,stats=4))
pympler.asizeof can be used to investigate how much memory certain Python objects consume.
In contrast to sys.getsizeof, asizeof sizes objects recursively
tracker from pympler
from pympler import tracker
tr = tracker.SummaryTracker()
def fun():
li = [1,2,3]
di = {"ha":"haha","duh":"Umm"}
fun()
tr.print_diff()
Tracks the lifetime of a function.
Pympler package consists of a huge number of high utility functions to profile code. All of which cannot be covered here. See the documentation attached for verbose profile implementations.
Pympler doc
Recently I created a plugin for PyCharm with which you can easily analyse and visualise the results of line_profiler
in the PyCharm editor.
line_profiler
has been mentioned in other answers as well and is a great tool to analyse exactly how much time is spent by the python interpreter in certain lines.
The PyCharm plugin I’ve created can be found here:
https://plugins.jetbrains.com/plugin/16536-line-profiler
It needs a helper package in your python environment called line-profiler-pycharm
which can be installed with pip or by the plugin itself.
After installing the plugin in PyCharm:
- Decorate any function you want to profile with the
line_profiler_pycharm.profile
decorator
- Run with the ‘Profile Lines’ runner
I found cprofiler and other ressources to be more for optimization purpose rather than debugging.
I made my own testing module instead for simple python scripts speed testing. (In my case 1K+ lines py file was tested using ScriptProfilerPy and speedup the code by 10x in minutes afterwards.
The module ScriptProfilerPy() will run your code adding timestamp to it.
I put the module here:
https://github.com/Lucas-BLP/ScriptProfilerPy
Use:
from speed_testpy import ScriptProfilerPy
ScriptProfilerPy("path_to_your_script_to_test.py").Profiler()
I find this function is quick and easy to use if you do not want a command line option.
To use just add @profile above each function to be profiled.
def profile(fnc):
"""
Profiles any function in following class just by adding @profile above function
"""
import cProfile, pstats, io
def inner (*args, **kwargs):
pr = cProfile.Profile()
pr.enable()
retval = fnc (*args, **kwargs)
pr.disable()
s = io.StringIO()
sortby = 'cumulative' #Ordered
ps = pstats.Stats(pr,stream=s).strip_dirs().sort_stats(sortby)
n=10 #reduced the list to be monitored
ps.print_stats(n)
#ps.dump_stats("profile.prof")
print(s.getvalue())
return retval
return inner
output for each function looks like this
Ordered by: cumulative time
List reduced from 38 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.002 0.002 3151212474.py:37(get_pdf_page_count)
1 0.000 0.000 0.002 0.002 fitz.py:3604(__init__)
1 0.001 0.001 0.001 0.001 {built-in method fitz._fitz.new_Document}
1 0.000 0.000 0.000 0.000 fitz.py:5207(__del__)
1 0.000 0.000 0.000 0.000 {built-in method fitz._fitz.delete_Document}
1 0.000 0.000 0.000 0.000 fitz.py:4816(init_doc)
1 0.000 0.000 0.000 0.000 fitz.py:5197(_reset_page_refs)
1 0.000 0.000 0.000 0.000 fitz.py:4821(<listcomp>)
11 0.000 0.000 0.000 0.000 fitz.py:4054(_getMetadata)
1 0.000 0.000 0.000 0.000 weakref.py:241(values)
Scalene is a new python profiler that covers many use cases and has a minimal performance impact:
https://github.com/plasma-umass/scalene
It can profile CPU, GPU and memory utilisation at a very granular level. It also notably supports multi-threaded / parallelized python code.
Project Euler and other coding contests often have a maximum time to run or people boast of how fast their particular solution runs. With Python, sometimes the approaches are somewhat kludgey – i.e., adding timing code to __main__
.
What is a good way to profile how long a Python program takes to run?
Python includes a profiler called cProfile. It not only gives the total running time, but also times each function separately, and tells you how many times each function was called, making it easy to determine where you should make optimizations.
You can call it from within your code, or from the interpreter, like this:
import cProfile
cProfile.run('foo()')
Even more usefully, you can invoke the cProfile when running a script:
python -m cProfile myscript.py
To make it even easier, I made a little batch file called ‘profile.bat’:
python -m cProfile %1
So all I have to do is run:
profile euler048.py
And I get this:
1007 function calls in 0.061 CPU seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.061 0.061 <string>:1(<module>)
1000 0.051 0.000 0.051 0.000 euler048.py:2(<lambda>)
1 0.005 0.005 0.061 0.061 euler048.py:2(<module>)
1 0.000 0.000 0.061 0.061 {execfile}
1 0.002 0.002 0.053 0.053 {map}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler objects}
1 0.000 0.000 0.000 0.000 {range}
1 0.003 0.003 0.003 0.003 {sum}
EDIT: Updated link to a good video resource from PyCon 2013 titled
Python Profiling
Also via YouTube.
In Virtaal’s source there’s a very useful class and decorator that can make profiling (even for specific methods/functions) very easy. The output can then be viewed very comfortably in KCacheGrind.
It’s worth pointing out that using the profiler only works (by default) on the main thread, and you won’t get any information from other threads if you use them. This can be a bit of a gotcha as it is completely unmentioned in the profiler documentation.
If you also want to profile threads, you’ll want to look at the threading.setprofile()
function in the docs.
You could also create your own threading.Thread
subclass to do it:
class ProfiledThread(threading.Thread):
# Overrides threading.Thread.run()
def run(self):
profiler = cProfile.Profile()
try:
return profiler.runcall(threading.Thread.run, self)
finally:
profiler.dump_stats('myprofile-%d.profile' % (self.ident,))
and use that ProfiledThread
class instead of the standard one. It might give you more flexibility, but I’m not sure it’s worth it, especially if you are using third-party code which wouldn’t use your class.
The python wiki is a great page for profiling resources:
http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code
as is the python docs:
http://docs.python.org/library/profile.html
as shown by Chris Lawlor cProfile is a great tool and can easily be used to print to the screen:
python -m cProfile -s time mine.py <args>
or to file:
python -m cProfile -o output.file mine.py <args>
PS> If you are using Ubuntu, make sure to install python-profile
apt-get install python-profiler
If you output to file you can get nice visualizations using the following tools
PyCallGraph : a tool to create call graph images
install:
pip install pycallgraph
run:
pycallgraph mine.py args
view:
gimp pycallgraph.png
You can use whatever you like to view the png file, I used gimp
Unfortunately I often get
dot: graph is too large for cairo-renderer bitmaps. Scaling by 0.257079 to fit
which makes my images unusably small. So I generally create svg files:
pycallgraph -f svg -o pycallgraph.svg mine.py <args>
PS> make sure to install graphviz (which provides the dot program):
pip install graphviz
Alternative Graphing using gprof2dot via @maxy / @quodlibetor :
pip install gprof2dot
python -m cProfile -o profile.pstats mine.py
gprof2dot -f pstats profile.pstats | dot -Tsvg -o mine.svg
A nice profiling module is the line_profiler (called using the script kernprof.py). It can be downloaded here.
My understanding is that cProfile only gives information about total time spent in each function. So individual lines of code are not timed. This is an issue in scientific computing since often one single line can take a lot of time. Also, as I remember, cProfile didn’t catch the time I was spending in say numpy.dot.
Following Joe Shaw’s answer about multi-threaded code not to work as expected, I figured that the runcall
method in cProfile is merely doing self.enable()
and self.disable()
calls around the profiled function call, so you can simply do that yourself and have whatever code you want in-between with minimal interference with existing code.
A while ago I made pycallgraph
which generates a visualisation from your Python code. Edit: I’ve updated the example to work with 3.3, the latest release as of this writing.
After a pip install pycallgraph
and installing GraphViz you can run it from the command line:
pycallgraph graphviz -- ./mypythonscript.py
Or, you can profile particular parts of your code:
from pycallgraph import PyCallGraph
from pycallgraph.output import GraphvizOutput
with PyCallGraph(output=GraphvizOutput()):
code_to_profile()
Either of these will generate a pycallgraph.png
file similar to the image below:
Ever want to know what the hell that python script is doing? Enter the
Inspect Shell. Inspect Shell lets you print/alter globals and run
functions without interrupting the running script. Now with
auto-complete and command history (only on linux).Inspect Shell is not a pdb-style debugger.
https://github.com/amoffat/Inspect-Shell
You could use that (and your wristwatch).
@Maxy’s comment on this answer helped me out enough that I think it deserves its own answer: I already had cProfile-generated .pstats files and I didn’t want to re-run things with pycallgraph, so I used gprof2dot, and got pretty svgs:
$ sudo apt-get install graphviz
$ git clone https://github.com/jrfonseca/gprof2dot
$ ln -s "$PWD"/gprof2dot/gprof2dot.py ~/bin
$ cd $PROJECT_DIR
$ gprof2dot.py -f pstats profile.pstats | dot -Tsvg -o callgraph.svg
and BLAM!
It uses dot (the same thing that pycallgraph uses) so output looks similar. I get the impression that gprof2dot loses less information though:
My way is to use yappi (https://github.com/sumerc/yappi). It’s especially useful combined with an RPC server where (even just for debugging) you register method to start, stop and print profiling information, e.g. in this way:
@staticmethod
def startProfiler():
yappi.start()
@staticmethod
def stopProfiler():
yappi.stop()
@staticmethod
def printProfiler():
stats = yappi.get_stats(yappi.SORTTYPE_TTOT, yappi.SORTORDER_DESC, 20)
statPrint = 'n'
namesArr = [len(str(stat[0])) for stat in stats.func_stats]
log.debug("namesArr %s", str(namesArr))
maxNameLen = max(namesArr)
log.debug("maxNameLen: %s", maxNameLen)
for stat in stats.func_stats:
nameAppendSpaces = [' ' for i in range(maxNameLen - len(stat[0]))]
log.debug('nameAppendSpaces: %s', nameAppendSpaces)
blankSpace = ''
for space in nameAppendSpaces:
blankSpace += space
log.debug("adding spaces: %s", len(nameAppendSpaces))
statPrint = statPrint + str(stat[0]) + blankSpace + " " + str(stat[1]).ljust(8) + "t" + str(
round(stat[2], 2)).ljust(8 - len(str(stat[2]))) + "t" + str(round(stat[3], 2)) + "n"
log.log(1000, "nname" + ''.ljust(maxNameLen - 4) + " ncall tttot ttsub")
log.log(1000, statPrint)
Then when your program work you can start profiler at any time by calling the startProfiler
RPC method and dump profiling information to a log file by calling printProfiler
(or modify the rpc method to return it to the caller) and get such output:
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
name ncall ttot tsub
2014-02-19 16:32:24,128-|SVR-MAIN |-(Thread-3 )-Level 1000:
C:Python27libsched.py.run:80 22 0.11 0.05
M: 2_documents_repos 9_aheadReposappsahdModbusSrvpyAheadRpcSrvxmlRpc.py.iterFnc:293 22 0.11 0.0
M: 2_documents_repos 9_aheadReposappsahdModbusSrvserverMain.py.makeIteration:515 22 0.11 0.0
M: 2_documents_repos 9_aheadReposappsahdModbusSrvpyAheadRpcSrvPicklingXMLRPC.py._dispatch:66 1 0.0 0.0
C:Python27libBaseHTTPServer.py.date_time_string:464 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil_psmswindows.py._get_raw_meminfo:243 4 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.decode_request_content:537 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil_psmswindows.py.get_system_cpu_times:148 4 0.0 0.0
<string>.__new__:8 220 0.0 0.0
C:Python27libsocket.py.close:276 4 0.0 0.0
C:Python27libthreading.py.__init__:558 1 0.0 0.0
<string>.__new__:8 4 0.0 0.0
C:Python27libthreading.py.notify:372 1 0.0 0.0
C:Python27librfc822.py.getheader:285 4 0.0 0.0
C:Python27libBaseHTTPServer.py.handle_one_request:301 1 0.0 0.0
C:Python27libxmlrpclib.py.end:816 3 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.do_POST:467 1 0.0 0.0
C:Python27libSimpleXMLRPCServer.py.is_rpc_path_valid:460 1 0.0 0.0
C:Python27libSocketServer.py.close_request:475 1 0.0 0.0
c:userszasiec~1appdatalocaltempeasy_install-hwcsr1psutil-1.1.2-py2.7-win32.egg.tmppsutil__init__.py.cpu_times:1066 4 0.0 0.0
It may not be very useful for short scripts but helps to optimize server-type processes especially given the printProfiler
method can be called multiple times over time to profile and compare e.g. different program usage scenarios.
In newer versions of yappi, the following code will work:
@staticmethod
def printProfile():
yappi.get_func_stats().print_all()
Also worth mentioning is the GUI cProfile dump viewer RunSnakeRun. It allows you to sort and select, thereby zooming in on the relevant parts of the program. The sizes of the rectangles in the picture is proportional to the time taken. If you mouse over a rectangle it highlights that call in the table and everywhere on the map. When you double-click on a rectangle it zooms in on that portion. It will show you who calls that portion and what that portion calls.
The descriptive information is very helpful. It shows you the code for that bit which can be helpful when you are dealing with built-in library calls. It tells you what file and what line to find the code.
Also want to point at that the OP said ‘profiling’ but it appears he meant ‘timing’. Keep in mind programs will run slower when profiled.
pprofile
line_profiler
(already presented here) also inspired pprofile
, which is described as:
Line-granularity, thread-aware deterministic and statistic pure-python
profiler
It provides line-granularity as line_profiler
, is pure Python, can be used as a standalone command or a module, and can even generate callgrind-format files that can be easily analyzed with [k|q]cachegrind
.
vprof
There is also vprof, a Python package described as:
[…] providing rich and interactive visualizations for various Python program characteristics such as running time and memory usage.
To add on to https://stackoverflow.com/a/582337/1070617,
I wrote this module that allows you to use cProfile and view its output easily. More here: https://github.com/ymichael/cprofilev
$ python -m cprofilev /your/python/program
# Go to http://localhost:4000 to view collected statistics.
Also see: http://ymichael.com/2014/03/08/profiling-python-with-cprofile.html on how to make sense of the collected statistics.
cProfile is great for quick profiling but most of the time it was ending for me with the errors. Function runctx solves this problem by initializing correctly the environment and variables, hope it can be useful for someone:
import cProfile
cProfile.runctx('foo()', None, locals())
A new tool to handle profiling in Python is PyVmMonitor: http://www.pyvmmonitor.com/
It has some unique features such as
- Attach profiler to a running (CPython) program
- On demand profiling with Yappi integration
- Profile on a different machine
- Multiple processes support (multiprocessing, django…)
- Live sampling/CPU view (with time range selection)
- Deterministic profiling through cProfile/profile integration
- Analyze existing PStats results
- Open DOT files
- Programatic API access
- Group samples by method or line
- PyDev integration
- PyCharm integration
Note: it’s commercial, but free for open source.
There’s a lot of great answers but they either use command line or some external program for profiling and/or sorting the results.
I really missed some way I could use in my IDE (eclipse-PyDev) without touching the command line or installing anything. So here it is.
Profiling without command line
def count():
from math import sqrt
for x in range(10**5):
sqrt(x)
if __name__ == '__main__':
import cProfile, pstats
cProfile.run("count()", "{}.profile".format(__file__))
s = pstats.Stats("{}.profile".format(__file__))
s.strip_dirs()
s.sort_stats("time").print_stats(10)
See docs or other answers for more info.
There’s also a statistical profiler called statprof
. It’s a sampling profiler, so it adds minimal overhead to your code and gives line-based (not just function-based) timings. It’s more suited to soft real-time applications like games, but may be have less precision than cProfile.
The version in pypi is a bit old, so can install it with pip
by specifying the git repository:
pip install git+git://github.com/bos/[email protected]
You can run it like this:
import statprof
with statprof.profile():
my_questionable_function()
cProfile
is great for profiling, while kcachegrind
is great for visualizing the results. The pyprof2calltree
in between handles the file conversion.
python -m cProfile -o script.profile script.py
pyprof2calltree -i script.profile -o script.calltree
kcachegrind script.calltree
Required system packages:
kcachegrind
(Linux),qcachegrind
(MacOs)
Setup on Ubuntu:
apt-get install kcachegrind
pip install pyprof2calltree
The result:
I ran into a handy tool called SnakeViz when researching this topic. SnakeViz is a web-based profiling visualization tool. It is very easy to install and use. The usual way I use it is to generate a stat file with %prun
and then do analysis in SnakeViz.
The main viz technique used is Sunburst chart as shown below, in which the hierarchy of function calls is arranged as layers of arcs and time info encoded in their angular widths.
The best thing is you can interact with the chart. For example, to zoom in one can click on an arc, and the arc and its descendants will be enlarged as a new sunburst to display more details.
When i’m not root on the server, I use
lsprofcalltree.py and run my program like this:
python lsprofcalltree.py -o callgrind.1 test.py
Then I can open the report with any callgrind-compatible software, like qcachegrind
It would depend on what you want to see out of profiling. Simple time
metrics can be given by (bash).
time python python_prog.py
Even ‘/usr/bin/time’ can output detailed metrics by using ‘–verbose’ flag.
To check time metrics given by each function and to better understand how much time is spent on functions, you can use the inbuilt cProfile in python.
Going into more detailed metrics like performance, time is not the only metric. You can worry about memory, threads etc.
Profiling options:
1. line_profiler is another profiler used commonly to find out timing metrics line-by-line.
2. memory_profiler is a tool to profile memory usage.
3. heapy (from project Guppy) Profile how objects in the heap are used.
These are some of the common ones I tend to use. But if you want to find out more, try reading this book
It is a pretty good book on starting out with performance in mind. You can move onto advanced topics on using Cython and JIT(Just-in-time) compiled python.
Simplest and quickest way to find where all the time is going.
1. pip install snakeviz
2. python -m cProfile -o temp.dat <PROGRAM>.py
3. snakeviz temp.dat
Draws a pie chart in a browser. Biggest piece is the problem function. Very simple.
I recently created tuna for visualizing Python runtime and import profiles; this may be helpful here.
Install with
pip install tuna
Create a runtime profile
python3 -m cProfile -o program.prof yourfile.py
or an import profile (Python 3.7+ required)
python3 -X importprofile yourfile.py 2> import.log
Then just run tuna on the file
tuna program.prof
gprof2dot_magic
Magic function for gprof2dot
to profile any Python statement as a DOT graph in JupyterLab or Jupyter Notebook.
GitHub repo: https://github.com/mattijn/gprof2dot_magic
installation
Make sure you’ve the Python package gprof2dot_magic
.
pip install gprof2dot_magic
Its dependencies gprof2dot
and graphviz
will be installed as well
usage
To enable the magic function, first load the gprof2dot_magic
module
%load_ext gprof2dot_magic
and then profile any line statement as a DOT graph as such:
%gprof2dot print('hello world')
The terminal-only (and simplest) solution, in case all those fancy UI’s fail to install or to run:
ignore cProfile
completely and replace it with pyinstrument
, that will collect and display the tree of calls right after execution.
Install:
$ pip install pyinstrument
Profile and display result:
$ python -m pyinstrument ./prog.py
Works with python2 and 3.
[EDIT]
The documentation of the API, for profiling only a part of the code, can be found here.
If you want to make a cumulative profiler,
meaning to run the function several times in a row and watch the sum of the results.
you can use this cumulative_profiler
decorator:
it’s python >= 3.6 specific, but you can remove nonlocal
for it work on older versions.
import cProfile, pstats
class _ProfileFunc:
def __init__(self, func, sort_stats_by):
self.func = func
self.profile_runs = []
self.sort_stats_by = sort_stats_by
def __call__(self, *args, **kwargs):
pr = cProfile.Profile()
pr.enable() # this is the profiling section
retval = self.func(*args, **kwargs)
pr.disable()
self.profile_runs.append(pr)
ps = pstats.Stats(*self.profile_runs).sort_stats(self.sort_stats_by)
return retval, ps
def cumulative_profiler(amount_of_times, sort_stats_by='time'):
def real_decorator(function):
def wrapper(*args, **kwargs):
nonlocal function, amount_of_times, sort_stats_by # for python 2.x remove this row
profiled_func = _ProfileFunc(function, sort_stats_by)
for i in range(amount_of_times):
retval, ps = profiled_func(*args, **kwargs)
ps.print_stats()
return retval # returns the results of the function
return wrapper
if callable(amount_of_times): # incase you don't want to specify the amount of times
func = amount_of_times # amount_of_times is the function in here
amount_of_times = 5 # the default amount
return real_decorator(func)
return real_decorator
Example
profiling the function baz
import time
@cumulative_profiler
def baz():
time.sleep(1)
time.sleep(2)
return 1
baz()
baz
ran 5 times and printed this:
20 function calls in 15.003 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
10 15.003 1.500 15.003 1.500 {built-in method time.sleep}
5 0.000 0.000 15.003 3.001 <ipython-input-9-c89afe010372>:3(baz)
5 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
specifying the amount of times
@cumulative_profiler(3)
def baz():
...
I just developed my own profiler inspired from pypref_time:
https://github.com/modaresimr/auto_profiler
Update Version 2
Install:
pip install auto_profiler
Quick Start:
from auto_profiler import Profiler
with Profiler():
your_function()
Using in Jupyter, let you have realtime view of elapsed times
Update Version 1
By adding a decorator it will show a tree of time-consuming functions
@Profiler(depth=4)
Install by: pip install auto_profiler
Example
import time # line number 1
import random
from auto_profiler import Profiler, Tree
def f1():
mysleep(.6+random.random())
def mysleep(t):
time.sleep(t)
def fact(i):
f1()
if(i==1):
return 1
return i*fact(i-1)
def main():
for i in range(5):
f1()
fact(3)
with Profiler(depth=4):
main()
Example Output
Time [Hits * PerHit] Function name [Called from] [function location]
-----------------------------------------------------------------------
8.974s [1 * 8.974] main [auto-profiler/profiler.py:267] [/test/t2.py:30]
├── 5.954s [5 * 1.191] f1 [/test/t2.py:34] [/test/t2.py:14]
│ └── 5.954s [5 * 1.191] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 5.954s [5 * 1.191] <time.sleep>
|
|
| # The rest is for the example recursive function call fact
└── 3.020s [1 * 3.020] fact [/test/t2.py:36] [/test/t2.py:20]
├── 0.849s [1 * 0.849] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 0.849s [1 * 0.849] mysleep [/test/t2.py:15] [/test/t2.py:17]
│ └── 0.849s [1 * 0.849] <time.sleep>
└── 2.171s [1 * 2.171] fact [/test/t2.py:24] [/test/t2.py:20]
├── 1.552s [1 * 1.552] f1 [/test/t2.py:21] [/test/t2.py:14]
│ └── 1.552s [1 * 1.552] mysleep [/test/t2.py:15] [/test/t2.py:17]
└── 0.619s [1 * 0.619] fact [/test/t2.py:24] [/test/t2.py:20]
└── 0.619s [1 * 0.619] f1 [/test/t2.py:21] [/test/t2.py:14]
With a statistical profiler like austin, no instrumentation is required, meaning that you can get profiling data out of a Python application simply with
austin python3 my_script.py
The raw output isn’t very useful, but you can pipe that to flamegraph.pl
to get a flame graph representation of that data that gives you a breakdown of where the time (measured in microseconds of real time) is being spent.
austin python3 my_script.py | flamegraph.pl > my_script_profile.svg
Alternatively, you can also use the web application Speedscope.app for quick visualisation of the collected samples. If you have pprof installed, you can also get austin-python (with e.g. pipx install austin-python
) and use the austin2pprof
to covert to the pprof format.
However, if you have VS Code installed you could use the Austin extension for a more interactive experience, with source code heat maps, top functions and collected call stacks
If you’d rather use the terminal, you can also use the TUI, that also has a live graph mode:
For getting quick profile stats on an IPython notebook.
One can embed line_profiler and memory_profiler straight into their notebooks.
Another useful package is Pympler. It is a powerful profiling package that’s capable to track classes,objects,functions,memory leaks etc. Examples below, Docs attached.
Get it!
!pip install line_profiler
!pip install memory_profiler
!pip install pympler
Load it!
%load_ext line_profiler
%load_ext memory_profiler
Use it!
%time
%time print('Outputs CPU time,Wall Clock time')
#CPU times: user 2 µs, sys: 0 ns, total: 2 µs Wall time: 5.96 µs
Gives:
- CPU times: CPU level execution time
- sys times: system level execution time
- total: CPU time + system time
- Wall time: Wall Clock Time
%timeit
%timeit -r 7 -n 1000 print('Outputs execution time of the snippet')
#1000 loops, best of 7: 7.46 ns per loop
- Gives best time out of given number of runs(r) in looping (n) times.
- Outputs details on system caching:
- When code snippets are executed multiple times, system caches a few opearations and doesn’t execute them again that may hamper the accuracy of the profile reports.
%prun
%prun -s cumulative 'Code to profile'
Gives:
- number of function calls(ncalls)
- has entries per function call(distinct)
- time taken per call(percall)
- time elapsed till that function call(cumtime)
- name of the func/module called etc…
%memit
%memit 'Code to profile'
#peak memory: 199.45 MiB, increment: 0.00 MiB
Gives:
- Memory usage
%lprun
#Example function
def fun():
for i in range(10):
print(i)
#Usage: %lprun <name_of_the_function> function
%lprun -f fun fun()
Gives:
- Line wise stats
sys.getsizeof
sys.getsizeof('code to profile')
# 64 bytes
Returns the size of an object in bytes.
asizeof() from pympler
from pympler import asizeof
obj = [1,2,("hey","ha"),3]
print(asizeof.asizeof(obj,stats=4))
pympler.asizeof can be used to investigate how much memory certain Python objects consume.
In contrast to sys.getsizeof, asizeof sizes objects recursively
tracker from pympler
from pympler import tracker
tr = tracker.SummaryTracker()
def fun():
li = [1,2,3]
di = {"ha":"haha","duh":"Umm"}
fun()
tr.print_diff()
Tracks the lifetime of a function.
Pympler package consists of a huge number of high utility functions to profile code. All of which cannot be covered here. See the documentation attached for verbose profile implementations.
Pympler doc
Recently I created a plugin for PyCharm with which you can easily analyse and visualise the results of line_profiler
in the PyCharm editor.
line_profiler
has been mentioned in other answers as well and is a great tool to analyse exactly how much time is spent by the python interpreter in certain lines.
The PyCharm plugin I’ve created can be found here:
https://plugins.jetbrains.com/plugin/16536-line-profiler
It needs a helper package in your python environment called line-profiler-pycharm
which can be installed with pip or by the plugin itself.
After installing the plugin in PyCharm:
- Decorate any function you want to profile with the
line_profiler_pycharm.profile
decorator - Run with the ‘Profile Lines’ runner
I found cprofiler and other ressources to be more for optimization purpose rather than debugging.
I made my own testing module instead for simple python scripts speed testing. (In my case 1K+ lines py file was tested using ScriptProfilerPy and speedup the code by 10x in minutes afterwards.
The module ScriptProfilerPy() will run your code adding timestamp to it.
I put the module here:
https://github.com/Lucas-BLP/ScriptProfilerPy
Use:
from speed_testpy import ScriptProfilerPy
ScriptProfilerPy("path_to_your_script_to_test.py").Profiler()
I find this function is quick and easy to use if you do not want a command line option.
To use just add @profile above each function to be profiled.
def profile(fnc):
"""
Profiles any function in following class just by adding @profile above function
"""
import cProfile, pstats, io
def inner (*args, **kwargs):
pr = cProfile.Profile()
pr.enable()
retval = fnc (*args, **kwargs)
pr.disable()
s = io.StringIO()
sortby = 'cumulative' #Ordered
ps = pstats.Stats(pr,stream=s).strip_dirs().sort_stats(sortby)
n=10 #reduced the list to be monitored
ps.print_stats(n)
#ps.dump_stats("profile.prof")
print(s.getvalue())
return retval
return inner
output for each function looks like this
Ordered by: cumulative time
List reduced from 38 to 10 due to restriction <10>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.002 0.002 3151212474.py:37(get_pdf_page_count)
1 0.000 0.000 0.002 0.002 fitz.py:3604(__init__)
1 0.001 0.001 0.001 0.001 {built-in method fitz._fitz.new_Document}
1 0.000 0.000 0.000 0.000 fitz.py:5207(__del__)
1 0.000 0.000 0.000 0.000 {built-in method fitz._fitz.delete_Document}
1 0.000 0.000 0.000 0.000 fitz.py:4816(init_doc)
1 0.000 0.000 0.000 0.000 fitz.py:5197(_reset_page_refs)
1 0.000 0.000 0.000 0.000 fitz.py:4821(<listcomp>)
11 0.000 0.000 0.000 0.000 fitz.py:4054(_getMetadata)
1 0.000 0.000 0.000 0.000 weakref.py:241(values)
Scalene is a new python profiler that covers many use cases and has a minimal performance impact:
https://github.com/plasma-umass/scalene
It can profile CPU, GPU and memory utilisation at a very granular level. It also notably supports multi-threaded / parallelized python code.