How do I use line_profiler (from Robert Kern)?
Question:
I have tried using the line_profiler module for getting a line-by-line profile over a Python file. This is what I’ve done so far:
1) Installed line_profiler from pypi by using the .exe file (I am on WinXP and Win7). Just clicked through the installation wizard.
2) Written a small piece of code (similar to what has been asked in another answered question here).
from line_profiler import LineProfiler
def do_stuff(numbers):
print numbers
numbers = 2
profile = LineProfiler(do_stuff(numbers))
profile.print_stats()
3) Run the code from IDLE/PyScripter. I got only the time.
Timer unit: 4.17188e-10 s
How do I get full line-by-line profile over the code I execute? I have never used any advanced Python features like decorators, so it is hard for me to understand how shall I use the guidelines provided by several posts like here and here.
Answers:
Just follow Dan Riti’s example from the first link, but use your code. All you have to do after installing the line_profiler module is add a @profile decorator right before each function you wish to profile line-by-line and make sure each one is called at least once somewhere else in the code—so for your trivial example code that would be something like this:
example.py file:
@profile
def do_stuff(numbers):
print numbers
numbers = 2
do_stuff(numbers)
Having done that, run your script via the kernprof.py✶ that was installed in your C:Python27Scripts directory. Here’s the (not very interesting) actual output from doing this in a Windows 7 command-line session:
> python "C:Python27Scriptskernprof.py" -l -v example.py
2
Wrote profile results to example.py.lprof
Timer unit: 3.2079e-07 s
File: example.py
Function: do_stuff at line 2
Total time: 0.00185256 s
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 @profile
2 def do_stuff(numbers):
3 1 5775 5775.0 100.0 print numbers
You likely need to adapt this last step—the running of your test script with kernprof.py instead of directly by the Python interpreter—in order to do the equivalent from within IDLE or PyScripter.
✶Update
It appears that in line_profiler v1.0, the kernprof utility is distributed as an executable, not a .py script file as it was when I wrote the above. This means the following now needs to used to invoke it from the command-line:
> "C:Python27Scriptskernprof.exe" -l -v example.py
This answer is a copy of my answer here for how to get line_profiler statistics from within a Python script (without using kernprof from the command line or having to add @profile decorators to functions and class methods). All answers (that I’ve seen) to similar line_profiler questions only describe using kernprof.
The line_profiler test cases (found on GitHub) have an example of how to generate profile data from within a Python script. You have to wrap the function that you want to profile and then call the wrapper passing any desired function arguments.
from line_profiler import LineProfiler
import random
def do_stuff(numbers):
s = sum(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
Output:
Timer unit: 1e-06 s
Total time: 0.000649 s
File: <ipython-input-2-2e060b054fea>
Function: do_stuff at line 4
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4 def do_stuff(numbers):
5 1 10 10.0 1.5 s = sum(numbers)
6 1 186 186.0 28.7 l = [numbers[i]/43 for i in range(len(numbers))]
7 1 453 453.0 69.8 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
Adding Additional Functions to Profile
Also, you can add additional functions to be profiled as well. For example, if you had a second called function and you only wrap the calling function, you’ll only see the profile results from the calling function.
from line_profiler import LineProfiler
import random
def do_other_stuff(numbers):
s = sum(numbers)
def do_stuff(numbers):
do_other_stuff(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
The above would only produce the following profile output for the calling function:
Timer unit: 1e-06 s
Total time: 0.000773 s
File: <ipython-input-3-ec0394d0a501>
Function: do_stuff at line 7
Line # Hits Time Per Hit % Time Line Contents
==============================================================
7 def do_stuff(numbers):
8 1 11 11.0 1.4 do_other_stuff(numbers)
9 1 236 236.0 30.5 l = [numbers[i]/43 for i in range(len(numbers))]
10 1 526 526.0 68.0 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
In this case, you can add the additional called function to profile like this:
from line_profiler import LineProfiler
import random
def do_other_stuff(numbers):
s = sum(numbers)
def do_stuff(numbers):
do_other_stuff(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(do_other_stuff) # add additional function to profile
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
Output:
Timer unit: 1e-06 s
Total time: 9e-06 s
File: <ipython-input-4-dae73707787c>
Function: do_other_stuff at line 4
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4 def do_other_stuff(numbers):
5 1 9 9.0 100.0 s = sum(numbers)
Total time: 0.000694 s
File: <ipython-input-4-dae73707787c>
Function: do_stuff at line 7
Line # Hits Time Per Hit % Time Line Contents
==============================================================
7 def do_stuff(numbers):
8 1 12 12.0 1.7 do_other_stuff(numbers)
9 1 208 208.0 30.0 l = [numbers[i]/43 for i in range(len(numbers))]
10 1 474 474.0 68.3 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
NOTE: Adding functions to profile in this way does not require changes to the profiled code (i.e., no need to add @profile decorators).
If you’re using PyCharm, you can also take a look at
https://plugins.jetbrains.com/plugin/16536-line-profiler
It’s a plugin I created that allows you to load and visualize line profiler results into the PyCharm editor.
load the line_profiler and numpy
%load_ext line_profiler
import numpy as np
define a function for example:
def take_sqr(array):
sqr_ar = [np.sqrt(x) for x in array]
return sqr_ar
use line_profiler to count the time as follows:
%lprun -f take_sqr take_sqr([1,2,3])
the output looks like this:
Timer unit: 1e-06 s
Total time: 6e-05 s File: <ipython-input-5-e50c1b05a473> Function:
take_sqr at line 1
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 def take_sqr(array):
2 4 59.0 14.8 98.3 sqr_ar = [np.sqrt(x) for x in array]
3 1 1.0 1.0 1.7 return sqr_ar
Found a good use to line_profiler using decorator i.e. @profile that worked for me:
def profile(func):
from functools import wraps
@wraps(func)
def wrapper(*args, **kwargs):
from line_profiler import LineProfiler
prof = LineProfiler()
try:
return prof(func)(*args, **kwargs)
finally:
prof.print_stats()
return wrapper
Credits to: pavelpatrin
Just an addition to @Lhenkel answer.
This is a decorator for async functions
def async_profile(func):
"""line profiler for an async funciton"""
from functools import wraps
@wraps(func)
async def wrapper(*args, **kwargs):
from line_profiler import LineProfiler
prof = LineProfiler()
try:
return await prof(func)(*args, **kwargs)
finally:
prof.print_stats()
return wrapper
To use these decorators with methods read this answer
I have tried using the line_profiler module for getting a line-by-line profile over a Python file. This is what I’ve done so far:
1) Installed line_profiler from pypi by using the .exe file (I am on WinXP and Win7). Just clicked through the installation wizard.
2) Written a small piece of code (similar to what has been asked in another answered question here).
from line_profiler import LineProfiler
def do_stuff(numbers):
print numbers
numbers = 2
profile = LineProfiler(do_stuff(numbers))
profile.print_stats()
3) Run the code from IDLE/PyScripter. I got only the time.
Timer unit: 4.17188e-10 s
How do I get full line-by-line profile over the code I execute? I have never used any advanced Python features like decorators, so it is hard for me to understand how shall I use the guidelines provided by several posts like here and here.
Just follow Dan Riti’s example from the first link, but use your code. All you have to do after installing the line_profiler module is add a @profile decorator right before each function you wish to profile line-by-line and make sure each one is called at least once somewhere else in the code—so for your trivial example code that would be something like this:
example.py file:
@profile
def do_stuff(numbers):
print numbers
numbers = 2
do_stuff(numbers)
Having done that, run your script via the kernprof.py✶ that was installed in your C:Python27Scripts directory. Here’s the (not very interesting) actual output from doing this in a Windows 7 command-line session:
> python "C:Python27Scriptskernprof.py" -l -v example.py
2
Wrote profile results to example.py.lprof
Timer unit: 3.2079e-07 s
File: example.py
Function: do_stuff at line 2
Total time: 0.00185256 s
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 @profile
2 def do_stuff(numbers):
3 1 5775 5775.0 100.0 print numbers
You likely need to adapt this last step—the running of your test script with kernprof.py instead of directly by the Python interpreter—in order to do the equivalent from within IDLE or PyScripter.
✶Update
It appears that in line_profiler v1.0, the kernprof utility is distributed as an executable, not a .py script file as it was when I wrote the above. This means the following now needs to used to invoke it from the command-line:
> "C:Python27Scriptskernprof.exe" -l -v example.py
This answer is a copy of my answer here for how to get line_profiler statistics from within a Python script (without using kernprof from the command line or having to add @profile decorators to functions and class methods). All answers (that I’ve seen) to similar line_profiler questions only describe using kernprof.
The line_profiler test cases (found on GitHub) have an example of how to generate profile data from within a Python script. You have to wrap the function that you want to profile and then call the wrapper passing any desired function arguments.
from line_profiler import LineProfiler
import random
def do_stuff(numbers):
s = sum(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
Output:
Timer unit: 1e-06 s
Total time: 0.000649 s
File: <ipython-input-2-2e060b054fea>
Function: do_stuff at line 4
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4 def do_stuff(numbers):
5 1 10 10.0 1.5 s = sum(numbers)
6 1 186 186.0 28.7 l = [numbers[i]/43 for i in range(len(numbers))]
7 1 453 453.0 69.8 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
Adding Additional Functions to Profile
Also, you can add additional functions to be profiled as well. For example, if you had a second called function and you only wrap the calling function, you’ll only see the profile results from the calling function.
from line_profiler import LineProfiler
import random
def do_other_stuff(numbers):
s = sum(numbers)
def do_stuff(numbers):
do_other_stuff(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
The above would only produce the following profile output for the calling function:
Timer unit: 1e-06 s
Total time: 0.000773 s
File: <ipython-input-3-ec0394d0a501>
Function: do_stuff at line 7
Line # Hits Time Per Hit % Time Line Contents
==============================================================
7 def do_stuff(numbers):
8 1 11 11.0 1.4 do_other_stuff(numbers)
9 1 236 236.0 30.5 l = [numbers[i]/43 for i in range(len(numbers))]
10 1 526 526.0 68.0 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
In this case, you can add the additional called function to profile like this:
from line_profiler import LineProfiler
import random
def do_other_stuff(numbers):
s = sum(numbers)
def do_stuff(numbers):
do_other_stuff(numbers)
l = [numbers[i]/43 for i in range(len(numbers))]
m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(do_other_stuff) # add additional function to profile
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()
Output:
Timer unit: 1e-06 s
Total time: 9e-06 s
File: <ipython-input-4-dae73707787c>
Function: do_other_stuff at line 4
Line # Hits Time Per Hit % Time Line Contents
==============================================================
4 def do_other_stuff(numbers):
5 1 9 9.0 100.0 s = sum(numbers)
Total time: 0.000694 s
File: <ipython-input-4-dae73707787c>
Function: do_stuff at line 7
Line # Hits Time Per Hit % Time Line Contents
==============================================================
7 def do_stuff(numbers):
8 1 12 12.0 1.7 do_other_stuff(numbers)
9 1 208 208.0 30.0 l = [numbers[i]/43 for i in range(len(numbers))]
10 1 474 474.0 68.3 m = ['hello'+str(numbers[i]) for i in range(len(numbers))]
NOTE: Adding functions to profile in this way does not require changes to the profiled code (i.e., no need to add @profile decorators).
If you’re using PyCharm, you can also take a look at
https://plugins.jetbrains.com/plugin/16536-line-profiler
It’s a plugin I created that allows you to load and visualize line profiler results into the PyCharm editor.
load the line_profiler and numpy
%load_ext line_profiler
import numpy as np
define a function for example:
def take_sqr(array):
sqr_ar = [np.sqrt(x) for x in array]
return sqr_ar
use line_profiler to count the time as follows:
%lprun -f take_sqr take_sqr([1,2,3])
the output looks like this:
Timer unit: 1e-06 s
Total time: 6e-05 s File: <ipython-input-5-e50c1b05a473> Function:
take_sqr at line 1
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 def take_sqr(array):
2 4 59.0 14.8 98.3 sqr_ar = [np.sqrt(x) for x in array]
3 1 1.0 1.0 1.7 return sqr_ar
Found a good use to line_profiler using decorator i.e. @profile that worked for me:
def profile(func):
from functools import wraps
@wraps(func)
def wrapper(*args, **kwargs):
from line_profiler import LineProfiler
prof = LineProfiler()
try:
return prof(func)(*args, **kwargs)
finally:
prof.print_stats()
return wrapper
Credits to: pavelpatrin
Just an addition to @Lhenkel answer.
This is a decorator for async functions
def async_profile(func):
"""line profiler for an async funciton"""
from functools import wraps
@wraps(func)
async def wrapper(*args, **kwargs):
from line_profiler import LineProfiler
prof = LineProfiler()
try:
return await prof(func)(*args, **kwargs)
finally:
prof.print_stats()
return wrapper
To use these decorators with methods read this answer