FastAPI with uvicorn won't allow thread usage to exceed ~65%
Question:
I have written a machine learning inference library which has python bindings. Under normal operation, this library will use 8 threads for inference, and will max out all 8 threads 100%. This is the desired behavior as the model is very heavy and I need to optimize for low latency (therefore I need to use all CPU resources).
If I write a python script and make calls to the inference function in this library (in an infinite loop), the 8 threads are maxed out as expected (here’s the output of the htop command).
Now here’s where I’m having an issue. I need to call this machine learning library from within a FastAPI server which I have written. I am using the following command from within my docker container to launch the FastAPI server CMD uvicorn main:app --host 0.0.0.0 --port 8080. As can be seen, I make use of uvicorn.
Now, here is where things get interesting. If I call the same inference function in the machine learning library, once again in an infinite loop, but this time from within one of my FastAPI endpoints, then the CPU usage is capped at ~65% per thread and won’t exceed this.
Any thoughts on why the CPU usage is being capped? I would like to allow it to hit 100% to make full use of the CPU. I am getting performance loss due to the CPU usage being capped.
Answers:
I was able to determine what the issue was.
I had defined my endpoints as def, and according to the FastAPI documentation, it will dispatch the request to a threadpool. This must have caused some sort of CPU contention, presumably related to the Python GIL.
Switching the endpoints to async def solved the problem and allowed each of the 8 threads to reach 100% CPU usage. It therefore reduced the latency by about 30%.
I have written a machine learning inference library which has python bindings. Under normal operation, this library will use 8 threads for inference, and will max out all 8 threads 100%. This is the desired behavior as the model is very heavy and I need to optimize for low latency (therefore I need to use all CPU resources).
If I write a python script and make calls to the inference function in this library (in an infinite loop), the 8 threads are maxed out as expected (here’s the output of the htop command).
Now here’s where I’m having an issue. I need to call this machine learning library from within a FastAPI server which I have written. I am using the following command from within my docker container to launch the FastAPI server CMD uvicorn main:app --host 0.0.0.0 --port 8080. As can be seen, I make use of uvicorn.
Now, here is where things get interesting. If I call the same inference function in the machine learning library, once again in an infinite loop, but this time from within one of my FastAPI endpoints, then the CPU usage is capped at ~65% per thread and won’t exceed this.
Any thoughts on why the CPU usage is being capped? I would like to allow it to hit 100% to make full use of the CPU. I am getting performance loss due to the CPU usage being capped.
I was able to determine what the issue was.
I had defined my endpoints as def, and according to the FastAPI documentation, it will dispatch the request to a threadpool. This must have caused some sort of CPU contention, presumably related to the Python GIL.
Switching the endpoints to async def solved the problem and allowed each of the 8 threads to reach 100% CPU usage. It therefore reduced the latency by about 30%.