Profiling a python program with PyCharm (or any other IDE)
Question:
I’m running a relatively complex python program and in it there is a montecarlo simulation which takes up most of the time. I would like to find out what part of it uses the most resources so I can potentially make it faster.
I’m using PyCharm Professional edition and tried to use the profiler, but the result is only a large list of irrelevant functions that I’ve never heard of.
Questions: Is there a good profiler I can use that delivers meaningful results so I can see which function or keyword uses the most resources in my montecarlo simulation?
Answers:
Depending on your needs and your python version, maybe you want to use something like hotshot. https://docs.python.org/2/library/hotshot.html
EDIT:
For python 3.4 cProfile is probably one the best options you have available but you will definitely have to filter the results with grep/sed/awk to be able to get the relevant results especially if you use libraries imported where there are a lot of internal calls occurring.
I like sorting by number of calls:
python -m cProfile -s 'calls' <your_program>.py
Now the issue in python3 with that method is the number of primitive calls that will show up if cProfile is called externally, so running it internally is probably a better idea:
import cProfile
pr = cProfile.Profile()
pr.enable()
your_function_call()
pr.disable()
# after your program ends
pr.print_stats(sort="calls")
Note: As mentioned in the comments, the following applies to the paid version of PyCharm:
If using 3.x (don’t know about 2.x), I’ll add to shafeen’s answer and make it more PyCharm specific as per the original post. This also works better for web applications or larger applications versus simple command line programs where printing the output to stdout might be okay (still better to be able to sort different ways through PyCharm’s viewer).
Indeed, do as suggested by instantiating Profile and enabling and disabling as needed. To make that useful though, you’ll want to save this to a file.
- In an outer section of your code, instantiate Profile.
- In the inner section of your code, do your profiling.
- Now, call pr.dump_stats(‘profile.pstat’)
You now have a profile file that you would like to examine. Go to Tools|Open CProfile snapshot. Select profile.pstat and now you can view and sort by different headings as desired.
Summary
import cProfile as profile
# In outer section of code
pr = profile.Profile()
pr.disable()
# In section you want to profile
pr.enable()
# code of interest
pr.disable()
# Back in outer section of code
pr.dump_stats('profile.pstat')
Open file in PyCharm’s CProfile viewer.
I am surprised nobody mentioned SnakeViz yet. Besides the profiler from Spyder this is the best python profiler I could find so far:
pip install snakeviz
Then:
python -m cProfile -o program.prof my_program.py
snakeviz program.prof
It will open some nice visualizations in your browser:
I’m running a relatively complex python program and in it there is a montecarlo simulation which takes up most of the time. I would like to find out what part of it uses the most resources so I can potentially make it faster.
I’m using PyCharm Professional edition and tried to use the profiler, but the result is only a large list of irrelevant functions that I’ve never heard of.
Questions: Is there a good profiler I can use that delivers meaningful results so I can see which function or keyword uses the most resources in my montecarlo simulation?
Depending on your needs and your python version, maybe you want to use something like hotshot. https://docs.python.org/2/library/hotshot.html
EDIT:
For python 3.4 cProfile is probably one the best options you have available but you will definitely have to filter the results with grep/sed/awk to be able to get the relevant results especially if you use libraries imported where there are a lot of internal calls occurring.
I like sorting by number of calls:
python -m cProfile -s 'calls' <your_program>.py
Now the issue in python3 with that method is the number of primitive calls that will show up if cProfile is called externally, so running it internally is probably a better idea:
import cProfile
pr = cProfile.Profile()
pr.enable()
your_function_call()
pr.disable()
# after your program ends
pr.print_stats(sort="calls")
Note: As mentioned in the comments, the following applies to the paid version of PyCharm:
If using 3.x (don’t know about 2.x), I’ll add to shafeen’s answer and make it more PyCharm specific as per the original post. This also works better for web applications or larger applications versus simple command line programs where printing the output to stdout might be okay (still better to be able to sort different ways through PyCharm’s viewer).
Indeed, do as suggested by instantiating Profile and enabling and disabling as needed. To make that useful though, you’ll want to save this to a file.
- In an outer section of your code, instantiate Profile.
- In the inner section of your code, do your profiling.
- Now, call pr.dump_stats(‘profile.pstat’)
You now have a profile file that you would like to examine. Go to Tools|Open CProfile snapshot. Select profile.pstat and now you can view and sort by different headings as desired.
Summary
import cProfile as profile
# In outer section of code
pr = profile.Profile()
pr.disable()
# In section you want to profile
pr.enable()
# code of interest
pr.disable()
# Back in outer section of code
pr.dump_stats('profile.pstat')
Open file in PyCharm’s CProfile viewer.
I am surprised nobody mentioned SnakeViz yet. Besides the profiler from Spyder this is the best python profiler I could find so far:
pip install snakeviz
Then:
python -m cProfile -o program.prof my_program.py
snakeviz program.prof
It will open some nice visualizations in your browser: