Architecture Flask vs FastAPI
Question:
I have been tinkering around Flask and FastAPI to see how it acts as a server.
One of the main things that I would like to know is how Flask and FastAPI deal with multiple requests from multiple clients.
Especially when the code has efficiency issues (long database query time).
So, I tried making a simple code to understand this problem.
The code is simple, when the client access the route, the application sleeps for 10 seconds before it returns results.
It looks something like this:
FastAPI
import uvicorn
from fastapi import FastAPI
from time import sleep
app = FastAPI()
@app.get('/')
async def root():
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
Flask
from flask import Flask
from flask_restful import Resource, Api
from time import sleep
app = Flask(__name__)
api = Api(app)
class Root(Resource):
def get(self):
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
api.add_resource(Root, '/')
if __name__ == "__main__":
app.run()
Once the applications are up, I tried accessing them at the same time through 2 different chrome clients.
The below are the results:
FastAPI
Flask
As you can see, for FastAPI, the code first waits 10 seconds before processing the next request. Whereas for Flask, the code processes the next request while the 10-second sleep is still happening.
Despite doing a bit of googling, there is not really a straight answer on this topic.
If anyone has any comments that can shed some light on this, please drop them in the comments.
Your opinions are all appreciated. Thank you all very much for your time.
EDIT
An update on this, I am exploring a bit more and found this concept of Process manager. For example, we can run uvicorn using a process manager (gunicorn). By adding more workers, I am able to achieve something like Flask. Still testing the limits of this, however.
https://www.uvicorn.org/deployment/
Thanks to everyone who left comments! Appreciate it.
Answers:
I think you are blocking an event queue in FastAPI which is asynchronous framework whereas in Flask requests are probably run each in new thread. Move all CPU bound tasks to separate processes or in your FastAPI example just sleep on event loop (do not use time.sleep here). In FastAPI run IO bound tasks asynchronously
This seemed a little interesting, so i ran a little tests with ApacheBench:
Flask
from flask import Flask
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class Root(Resource):
def get(self):
return {"message": "hello"}
api.add_resource(Root, "/")
FastAPI
from fastapi import FastAPI
app = FastAPI(debug=False)
@app.get("/")
async def root():
return {"message": "hello"}
I ran 2 tests for FastAPI, there was a huge difference:
gunicorn -w 4 -k uvicorn.workers.UvicornWorker fast_api:appuvicorn fast_api:app --reload
So here is the benchmarking results for 5000 requests with a concurrency of 500:
FastAPI with Uvicorn Workers
Concurrency Level: 500
Time taken for tests: 0.577 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 8665.48 [#/sec] (mean)
Time per request: 57.700 [ms] (mean)
Time per request: 0.115 [ms] (mean, across all concurrent requests)
Transfer rate: 1218.58 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 4.5 6 30
Processing: 6 49 21.7 45 126
Waiting: 1 42 19.0 39 124
Total: 12 56 21.8 53 127
Percentage of the requests served within a certain time (ms)
50% 53
66% 64
75% 69
80% 73
90% 81
95% 98
98% 112
99% 116
100% 127 (longest request)
FastAPI – Pure Uvicorn
Concurrency Level: 500
Time taken for tests: 1.562 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 3200.62 [#/sec] (mean)
Time per request: 156.220 [ms] (mean)
Time per request: 0.312 [ms] (mean, across all concurrent requests)
Transfer rate: 450.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 8 4.8 7 24
Processing: 26 144 13.1 143 195
Waiting: 2 132 13.1 130 181
Total: 26 152 12.6 150 203
Percentage of the requests served within a certain time (ms)
50% 150
66% 155
75% 158
80% 160
90% 166
95% 171
98% 195
99% 199
100% 203 (longest request)
For Flask:
Concurrency Level: 500
Time taken for tests: 27.827 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 830000 bytes
HTML transferred: 105000 bytes
Requests per second: 179.68 [#/sec] (mean)
Time per request: 2782.653 [ms] (mean)
Time per request: 5.565 [ms] (mean, across all concurrent requests)
Transfer rate: 29.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 87 293.2 0 3047
Processing: 14 1140 4131.5 136 26794
Waiting: 1 1140 4131.5 135 26794
Total: 14 1227 4359.9 136 27819
Percentage of the requests served within a certain time (ms)
50% 136
66% 148
75% 179
80% 198
90% 295
95% 7839
98% 14518
99% 27765
100% 27819 (longest request)
Total results
Flask: Time taken for tests: 27.827 seconds
FastAPI – Uvicorn: Time taken for tests: 1.562 seconds
FastAPI – Uvicorn Workers: Time taken for tests: 0.577 seconds
With Uvicorn Workers FastAPI is nearly 48x faster than Flask, which is very understandable. ASGI vs WSGI, so i ran with 1 concurreny:
FastAPI – UvicornWorkers: Time taken for tests: 1.615 seconds
FastAPI – Pure Uvicorn: Time taken for tests: 2.681 seconds
Flask: Time taken for tests: 5.541 seconds
I ran more tests to test out Flask with a production server.
5000 Request 1000 Concurrency
Flask with Waitress
Server Software: waitress
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 21 bytes
Concurrency Level: 1000
Time taken for tests: 3.403 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 830000 bytes
HTML transferred: 105000 bytes
Requests per second: 1469.47 [#/sec] (mean)
Time per request: 680.516 [ms] (mean)
Time per request: 0.681 [ms] (mean, across all concurrent requests)
Transfer rate: 238.22 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 8.6 0 30
Processing: 31 607 156.3 659 754
Waiting: 1 607 156.3 658 753
Total: 31 611 148.4 660 754
Percentage of the requests served within a certain time (ms)
50% 660
66% 678
75% 685
80% 691
90% 702
95% 728
98% 743
99% 750
100% 754 (longest request)
Gunicorn with Uvicorn Workers
Server Software: uvicorn
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 19 bytes
Concurrency Level: 1000
Time taken for tests: 0.634 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 7891.28 [#/sec] (mean)
Time per request: 126.722 [ms] (mean)
Time per request: 0.127 [ms] (mean, across all concurrent requests)
Transfer rate: 1109.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 28 13.8 30 62
Processing: 18 89 35.6 86 203
Waiting: 1 75 33.3 70 171
Total: 20 118 34.4 116 243
Percentage of the requests served within a certain time (ms)
50% 116
66% 126
75% 133
80% 137
90% 161
95% 189
98% 217
99% 230
100% 243 (longest request)
Pure Uvicorn, but this time 4 workers uvicorn fastapi:app --workers 4
Server Software: uvicorn
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 19 bytes
Concurrency Level: 1000
Time taken for tests: 1.147 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 4359.68 [#/sec] (mean)
Time per request: 229.375 [ms] (mean)
Time per request: 0.229 [ms] (mean, across all concurrent requests)
Transfer rate: 613.08 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 20 16.3 17 70
Processing: 17 190 96.8 171 501
Waiting: 3 173 93.0 151 448
Total: 51 210 96.4 184 533
Percentage of the requests served within a certain time (ms)
50% 184
66% 209
75% 241
80% 260
90% 324
95% 476
98% 504
99% 514
100% 533 (longest request)
You are using the time.sleep() function, in a async endpoint. time.sleep() is blocking and should never be used in asynchronous code. What you should be using is probably the asyncio.sleep() function:
import asyncio
import uvicorn
from fastapi import FastAPI
app = FastAPI()
@app.get('/')
async def root():
print('Sleeping for 10')
await asyncio.sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
That way, each request will take ~10 sec to complete, but you will be able to serve multiple requests concurrently.
In general, async frameworks offer replacements for all blocking functions inside the standard library (sleep functions, IO functions, etc.). You are meant to use those replacements when writing async code and (optionally) await them.
Some non-blocking frameworks and libraries such as gevent, do not offer replacements. They instead monkey-patch functions in the standard library to make them non-blocking. This is not the case, as far as I know, for the newer async frameworks and libraries though, because they are meant to allow the developer to use the async-await syntax.
Why code is slow
Blocking operations will stop your event loop running the tasks. When you are calling the sleep() function, all the tasks (requests) are waiting until it’s finished, thus killing all the benefits of asynchronous code execution.
To understand why this code is wrong for comparison, we should better understand how asynchronous code works in Python and have some knowledge of GIL. Concurrency and async code are well explained in the docs of FastAPI.
@Asotos has described why your code is slow and yes, you should use coroutines for I/O operations since they don’t block the event loop from execution (time.sleep() is a blocking I/O operation, but coroutine asyncio.sleep() is a non-blocking I/O operation). It is reasonably suggested to use async functions so that the event loop is not blocked, but for now, not all libraries have async versions.
Optimization without async functions and asyncio.sleep
In case you cannot use the async version of the library, you can simply define your route functions as simple def functions, not async def.
If the route function is defined as synchronous (def), FastAPI will smartly call this function in an external thread pool, and the main thread with event loop will not be blocked, and your benchmarks will be much better without using await asyncio.sleep(). Greatly explained in this section.
Solution
from time import sleep
import uvicorn
from fastapi import FastAPI
app = FastAPI()
@app.get('/')
def root():
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
BTW, you won’t gain a lot of benefits, if operations run in the thread pool are CPU bound (e.g. heavy calculations) because of GIL. CPU-bound tasks must be run in separate processes.
What actually need to do in FastAPI is actually using Background Tasks in real scenario which you may sending email or doing heavy query on database and something like that
FastAPI
from fastapi import FastAPI, BackgroundTasks
import time
app = FastAPI()
def sleep(msg):
time.sleep(10)
print(msg)
@app.get('/')
async def root(background_tasks: BackgroundTasks):
msg= 'Sleeping for 10'
background_tasks.add_task(sleep, msg)
print('Awake')
return {'message': 'hello'}
then try check benchmarks
I have been tinkering around Flask and FastAPI to see how it acts as a server.
One of the main things that I would like to know is how Flask and FastAPI deal with multiple requests from multiple clients.
Especially when the code has efficiency issues (long database query time).
So, I tried making a simple code to understand this problem.
The code is simple, when the client access the route, the application sleeps for 10 seconds before it returns results.
It looks something like this:
FastAPI
import uvicorn
from fastapi import FastAPI
from time import sleep
app = FastAPI()
@app.get('/')
async def root():
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
Flask
from flask import Flask
from flask_restful import Resource, Api
from time import sleep
app = Flask(__name__)
api = Api(app)
class Root(Resource):
def get(self):
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
api.add_resource(Root, '/')
if __name__ == "__main__":
app.run()
Once the applications are up, I tried accessing them at the same time through 2 different chrome clients.
The below are the results:
FastAPI
Flask
As you can see, for FastAPI, the code first waits 10 seconds before processing the next request. Whereas for Flask, the code processes the next request while the 10-second sleep is still happening.
Despite doing a bit of googling, there is not really a straight answer on this topic.
If anyone has any comments that can shed some light on this, please drop them in the comments.
Your opinions are all appreciated. Thank you all very much for your time.
EDIT
An update on this, I am exploring a bit more and found this concept of Process manager. For example, we can run uvicorn using a process manager (gunicorn). By adding more workers, I am able to achieve something like Flask. Still testing the limits of this, however.
https://www.uvicorn.org/deployment/
Thanks to everyone who left comments! Appreciate it.
I think you are blocking an event queue in FastAPI which is asynchronous framework whereas in Flask requests are probably run each in new thread. Move all CPU bound tasks to separate processes or in your FastAPI example just sleep on event loop (do not use time.sleep here). In FastAPI run IO bound tasks asynchronously
This seemed a little interesting, so i ran a little tests with ApacheBench:
Flask
from flask import Flask
from flask_restful import Resource, Api
app = Flask(__name__)
api = Api(app)
class Root(Resource):
def get(self):
return {"message": "hello"}
api.add_resource(Root, "/")
FastAPI
from fastapi import FastAPI
app = FastAPI(debug=False)
@app.get("/")
async def root():
return {"message": "hello"}
I ran 2 tests for FastAPI, there was a huge difference:
gunicorn -w 4 -k uvicorn.workers.UvicornWorker fast_api:appuvicorn fast_api:app --reload
So here is the benchmarking results for 5000 requests with a concurrency of 500:
FastAPI with Uvicorn Workers
Concurrency Level: 500
Time taken for tests: 0.577 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 8665.48 [#/sec] (mean)
Time per request: 57.700 [ms] (mean)
Time per request: 0.115 [ms] (mean, across all concurrent requests)
Transfer rate: 1218.58 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 6 4.5 6 30
Processing: 6 49 21.7 45 126
Waiting: 1 42 19.0 39 124
Total: 12 56 21.8 53 127
Percentage of the requests served within a certain time (ms)
50% 53
66% 64
75% 69
80% 73
90% 81
95% 98
98% 112
99% 116
100% 127 (longest request)
FastAPI – Pure Uvicorn
Concurrency Level: 500
Time taken for tests: 1.562 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 3200.62 [#/sec] (mean)
Time per request: 156.220 [ms] (mean)
Time per request: 0.312 [ms] (mean, across all concurrent requests)
Transfer rate: 450.09 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 8 4.8 7 24
Processing: 26 144 13.1 143 195
Waiting: 2 132 13.1 130 181
Total: 26 152 12.6 150 203
Percentage of the requests served within a certain time (ms)
50% 150
66% 155
75% 158
80% 160
90% 166
95% 171
98% 195
99% 199
100% 203 (longest request)
For Flask:
Concurrency Level: 500
Time taken for tests: 27.827 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 830000 bytes
HTML transferred: 105000 bytes
Requests per second: 179.68 [#/sec] (mean)
Time per request: 2782.653 [ms] (mean)
Time per request: 5.565 [ms] (mean, across all concurrent requests)
Transfer rate: 29.13 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 87 293.2 0 3047
Processing: 14 1140 4131.5 136 26794
Waiting: 1 1140 4131.5 135 26794
Total: 14 1227 4359.9 136 27819
Percentage of the requests served within a certain time (ms)
50% 136
66% 148
75% 179
80% 198
90% 295
95% 7839
98% 14518
99% 27765
100% 27819 (longest request)
Total results
Flask: Time taken for tests: 27.827 seconds
FastAPI – Uvicorn: Time taken for tests: 1.562 seconds
FastAPI – Uvicorn Workers: Time taken for tests: 0.577 seconds
With Uvicorn Workers FastAPI is nearly 48x faster than Flask, which is very understandable. ASGI vs WSGI, so i ran with 1 concurreny:
FastAPI – UvicornWorkers: Time taken for tests: 1.615 seconds
FastAPI – Pure Uvicorn: Time taken for tests: 2.681 seconds
Flask: Time taken for tests: 5.541 seconds
I ran more tests to test out Flask with a production server.
5000 Request 1000 Concurrency
Flask with Waitress
Server Software: waitress
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 21 bytes
Concurrency Level: 1000
Time taken for tests: 3.403 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 830000 bytes
HTML transferred: 105000 bytes
Requests per second: 1469.47 [#/sec] (mean)
Time per request: 680.516 [ms] (mean)
Time per request: 0.681 [ms] (mean, across all concurrent requests)
Transfer rate: 238.22 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 4 8.6 0 30
Processing: 31 607 156.3 659 754
Waiting: 1 607 156.3 658 753
Total: 31 611 148.4 660 754
Percentage of the requests served within a certain time (ms)
50% 660
66% 678
75% 685
80% 691
90% 702
95% 728
98% 743
99% 750
100% 754 (longest request)
Gunicorn with Uvicorn Workers
Server Software: uvicorn
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 19 bytes
Concurrency Level: 1000
Time taken for tests: 0.634 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 7891.28 [#/sec] (mean)
Time per request: 126.722 [ms] (mean)
Time per request: 0.127 [ms] (mean, across all concurrent requests)
Transfer rate: 1109.71 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 28 13.8 30 62
Processing: 18 89 35.6 86 203
Waiting: 1 75 33.3 70 171
Total: 20 118 34.4 116 243
Percentage of the requests served within a certain time (ms)
50% 116
66% 126
75% 133
80% 137
90% 161
95% 189
98% 217
99% 230
100% 243 (longest request)
Pure Uvicorn, but this time 4 workers uvicorn fastapi:app --workers 4
Server Software: uvicorn
Server Hostname: 127.0.0.1
Server Port: 8000
Document Path: /
Document Length: 19 bytes
Concurrency Level: 1000
Time taken for tests: 1.147 seconds
Complete requests: 5000
Failed requests: 0
Total transferred: 720000 bytes
HTML transferred: 95000 bytes
Requests per second: 4359.68 [#/sec] (mean)
Time per request: 229.375 [ms] (mean)
Time per request: 0.229 [ms] (mean, across all concurrent requests)
Transfer rate: 613.08 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 20 16.3 17 70
Processing: 17 190 96.8 171 501
Waiting: 3 173 93.0 151 448
Total: 51 210 96.4 184 533
Percentage of the requests served within a certain time (ms)
50% 184
66% 209
75% 241
80% 260
90% 324
95% 476
98% 504
99% 514
100% 533 (longest request)
You are using the time.sleep() function, in a async endpoint. time.sleep() is blocking and should never be used in asynchronous code. What you should be using is probably the asyncio.sleep() function:
import asyncio
import uvicorn
from fastapi import FastAPI
app = FastAPI()
@app.get('/')
async def root():
print('Sleeping for 10')
await asyncio.sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
That way, each request will take ~10 sec to complete, but you will be able to serve multiple requests concurrently.
In general, async frameworks offer replacements for all blocking functions inside the standard library (sleep functions, IO functions, etc.). You are meant to use those replacements when writing async code and (optionally) await them.
Some non-blocking frameworks and libraries such as gevent, do not offer replacements. They instead monkey-patch functions in the standard library to make them non-blocking. This is not the case, as far as I know, for the newer async frameworks and libraries though, because they are meant to allow the developer to use the async-await syntax.
Why code is slow
Blocking operations will stop your event loop running the tasks. When you are calling the sleep() function, all the tasks (requests) are waiting until it’s finished, thus killing all the benefits of asynchronous code execution.
To understand why this code is wrong for comparison, we should better understand how asynchronous code works in Python and have some knowledge of GIL. Concurrency and async code are well explained in the docs of FastAPI.
@Asotos has described why your code is slow and yes, you should use coroutines for I/O operations since they don’t block the event loop from execution (time.sleep() is a blocking I/O operation, but coroutine asyncio.sleep() is a non-blocking I/O operation). It is reasonably suggested to use async functions so that the event loop is not blocked, but for now, not all libraries have async versions.
Optimization without async functions and asyncio.sleep
In case you cannot use the async version of the library, you can simply define your route functions as simple def functions, not async def.
If the route function is defined as synchronous (def), FastAPI will smartly call this function in an external thread pool, and the main thread with event loop will not be blocked, and your benchmarks will be much better without using await asyncio.sleep(). Greatly explained in this section.
Solution
from time import sleep
import uvicorn
from fastapi import FastAPI
app = FastAPI()
@app.get('/')
def root():
print('Sleeping for 10')
sleep(10)
print('Awake')
return {'message': 'hello'}
if __name__ == "__main__":
uvicorn.run(app, host="127.0.0.1", port=8000)
BTW, you won’t gain a lot of benefits, if operations run in the thread pool are CPU bound (e.g. heavy calculations) because of GIL. CPU-bound tasks must be run in separate processes.
What actually need to do in FastAPI is actually using Background Tasks in real scenario which you may sending email or doing heavy query on database and something like that
FastAPI
from fastapi import FastAPI, BackgroundTasks
import time
app = FastAPI()
def sleep(msg):
time.sleep(10)
print(msg)
@app.get('/')
async def root(background_tasks: BackgroundTasks):
msg= 'Sleeping for 10'
background_tasks.add_task(sleep, msg)
print('Awake')
return {'message': 'hello'}
then try check benchmarks