Python multiprocessing in flask
Question:
This question has probably been asked, and more than likely answered, but I don’t know where to find it here.
Problem: I have a router for python’s flask, that takes a while to process data for each call. I need to make each of the calls to the routes be a thread in itself so it doesn’t have to wait for the requests to be loaded.
Answers:
Flask comes with a built-in development web server, but you shouldn’t be using it in production.
To get cool features like separate processes for each request and static file serving, you need to run an actual web service and a WSGI service in front of your Flask application.
The Flask docs provide several examples on how to set that up. Popular Web Server/WSGI combinations are Apache/mod_wsgi and Nginx/Gunicorn, but there are many other options.
A really good way of setting this up would be to use “uwsgi” as your application server (and protocol) and Nginx as your front-end proxy. These are super quick, scalable, handle threading, and it is one of the Flask-recommended methods. Though the flask documentation provides the basic config, this guide is one I’ve used and it gives a much-more in-depth walkthrough for installation. They are using Ubuntu, but with minor changes (to the install commands) it will work on most Linux flavors.
As others have mentioned, there are specialized tools to do this (celery seems to be the best), but if someone just wants to quickly get something set up and working, here is my approach which only uses Python’s multiprocessing module:
from flask import Flask
from multiprocessing import Process
import time
app = Flask(__name__)
def detachedProcessFunction(wait_time):
i=0
while i<wait_time:
i = i+1
print "loop running %d" % i
time.sleep(1)
@app.route('/start')
def start():
global p
p = Process(target=detachedProcessFunction, args=(15))
p.start()
return render_template('layout.html')
if __name__ == '__main__':
app.run(debug=True)
NOTE: This method won’t work for running object functions (e.g. obj.objFunc()). You will get an EOFError: Ran out of input inside a class. In this case, you should create the object inside a non-object/standalone function and pass the arguments you need to create the object. For more info see here
This question has probably been asked, and more than likely answered, but I don’t know where to find it here.
Problem: I have a router for python’s flask, that takes a while to process data for each call. I need to make each of the calls to the routes be a thread in itself so it doesn’t have to wait for the requests to be loaded.
Flask comes with a built-in development web server, but you shouldn’t be using it in production.
To get cool features like separate processes for each request and static file serving, you need to run an actual web service and a WSGI service in front of your Flask application.
The Flask docs provide several examples on how to set that up. Popular Web Server/WSGI combinations are Apache/mod_wsgi and Nginx/Gunicorn, but there are many other options.
A really good way of setting this up would be to use “uwsgi” as your application server (and protocol) and Nginx as your front-end proxy. These are super quick, scalable, handle threading, and it is one of the Flask-recommended methods. Though the flask documentation provides the basic config, this guide is one I’ve used and it gives a much-more in-depth walkthrough for installation. They are using Ubuntu, but with minor changes (to the install commands) it will work on most Linux flavors.
As others have mentioned, there are specialized tools to do this (celery seems to be the best), but if someone just wants to quickly get something set up and working, here is my approach which only uses Python’s multiprocessing module:
from flask import Flask
from multiprocessing import Process
import time
app = Flask(__name__)
def detachedProcessFunction(wait_time):
i=0
while i<wait_time:
i = i+1
print "loop running %d" % i
time.sleep(1)
@app.route('/start')
def start():
global p
p = Process(target=detachedProcessFunction, args=(15))
p.start()
return render_template('layout.html')
if __name__ == '__main__':
app.run(debug=True)
NOTE: This method won’t work for running object functions (e.g. obj.objFunc()). You will get an EOFError: Ran out of input inside a class. In this case, you should create the object inside a non-object/standalone function and pass the arguments you need to create the object. For more info see here