practice with threads in python
Question:
I know that Python has a global lock and i’ve read Glyph’s explaination of python multithreading. But I still want to try it out. What I decided to do as an easy (conceptually) task was to do horizontal and vertical edge detection on a picture.
Here’s what’s happening (pseudocode):
for pixels in picture:
apply sobel operator horizontal
for pixels in picture:
apply sobel operator vertical
These two loops can run completely independent of each other, and so would be prime candidates for multithreading. (running these two loops on any significantly large picture can take 10+ seconds). However, when I have tried to use the threading module in python, it takes twice as long because of the global lock. My question is should I abandon all hope of doing this in two threads in python and try in another language? If i can forge ahead, what module(s) should I use? If not, what language should I experiment in?
Answers:
If the sobel operator is CPU-bound, then you won’t get any benefit from multiple threads because python does not take advantage of multiple cores.
Conceivably you could spin off multiple processes, though I’m not sure if that would be practical for working on a single image.
10 seconds doesn’t seem like a lot of time to waste. If you’re concerned about time because you’ll be processing many images, then it might be easier to run multiple processes and have each process deal with a separate subset of the images.
Python 2.6 now includes the mulitprocessing module (formerly processing module on older versions of Python).
It has essentially the same interface as the threading module, but launches the execution into separate processes rather than threads. This allows Python to take advantage of multiple cores/CPUs and scales well for CPU-intensive tasks compared to the threading module approach.
Bulk matrix operations like the Sobel operator will definitely realize significant speed gains by (correctly) using Matlab/Octave. It is possible that NumPy may provide similar speedups for matrix/array ops.
I recommend using NumPy as well. Not only will it probably be faster, but if you use threads with it, there won’t be a global lock.
I’ll also suggest using multiprocessing as Jay suggests.
Anyways, if you really want to practice threading, I’d suggest playing around with PThreads in C. PThreads are insanely simple to use for basic cases and used all over the place.
Python mutliprocessing is the right choice if you want to practice parallel programming with Python. If you don’t have Python 2.6 (which you don’t if you’re using Ubuntu for example), you can use the Google code backported version of multiprocessing. It is part of PyPI, which means you can easily install it using EasyInstall (which is part of the python-setuptools package in Ubuntu).
I know that Python has a global lock and i’ve read Glyph’s explaination of python multithreading. But I still want to try it out. What I decided to do as an easy (conceptually) task was to do horizontal and vertical edge detection on a picture.
Here’s what’s happening (pseudocode):
for pixels in picture:
apply sobel operator horizontal
for pixels in picture:
apply sobel operator vertical
These two loops can run completely independent of each other, and so would be prime candidates for multithreading. (running these two loops on any significantly large picture can take 10+ seconds). However, when I have tried to use the threading module in python, it takes twice as long because of the global lock. My question is should I abandon all hope of doing this in two threads in python and try in another language? If i can forge ahead, what module(s) should I use? If not, what language should I experiment in?
If the sobel operator is CPU-bound, then you won’t get any benefit from multiple threads because python does not take advantage of multiple cores.
Conceivably you could spin off multiple processes, though I’m not sure if that would be practical for working on a single image.
10 seconds doesn’t seem like a lot of time to waste. If you’re concerned about time because you’ll be processing many images, then it might be easier to run multiple processes and have each process deal with a separate subset of the images.
Python 2.6 now includes the mulitprocessing module (formerly processing module on older versions of Python).
It has essentially the same interface as the threading module, but launches the execution into separate processes rather than threads. This allows Python to take advantage of multiple cores/CPUs and scales well for CPU-intensive tasks compared to the threading module approach.
Bulk matrix operations like the Sobel operator will definitely realize significant speed gains by (correctly) using Matlab/Octave. It is possible that NumPy may provide similar speedups for matrix/array ops.
I recommend using NumPy as well. Not only will it probably be faster, but if you use threads with it, there won’t be a global lock.
I’ll also suggest using multiprocessing as Jay suggests.
Anyways, if you really want to practice threading, I’d suggest playing around with PThreads in C. PThreads are insanely simple to use for basic cases and used all over the place.
Python mutliprocessing is the right choice if you want to practice parallel programming with Python. If you don’t have Python 2.6 (which you don’t if you’re using Ubuntu for example), you can use the Google code backported version of multiprocessing. It is part of PyPI, which means you can easily install it using EasyInstall (which is part of the python-setuptools package in Ubuntu).