Celery in python to build microservices
Question:
I want to break down a project to small microservices. I’ve been thinking of different tools like gRPC, but I think celery might be a better option for me. Please if I’m wrong and celery is not a good choice, tell me why.
What exactly I want to do?
- For example, I have three different servers, A, B and C. "A" must be able to send a request to "B". "B" must be listening all the time.
- "B" get request from "A", works on it, and then it must be able to send request to "C".
- "C" does some stuff with the given request from "B" and then it must return the result to "B".
- "B" works on the result and when it’s finished, it must return the result to "A".
What have I done?
- I created two RabbitMQ servers, one for "A" & "B" and another one for "B" and "C".
- I used send_task to send tasks to servers. One task to send a string from "A" to "B". "B" is already listening and waiting for requests from "A". There’s another send_task on "B" which sends a request to "C". "C" is also already running and waiting for requests from "B".
- I used backend=’rpc’ to return results.
When a request is sent by "A", "B" receives the request and send_task to "C". "C" also receives the request with no problem. The problem is returning results.
I tried different things, but I mostly got runtimeError: Never call result.get() within a task Celery error.
Does anyone have any solution?
Answers:
This particular error indicates that using result.get() within a task is a bad practice that could lead to deadlocks.
If you know what you are doing you can try allow_join_result but that could cause performance issues:
with allow_join_result():
task.get()
You can also wait until the task is succeed:
timeout = 0.1
while not result.ready():
time.sleep(timeout)
ret = result.get()
I would rethink the whole thing if I were you.
First, I think there is no reason for multiple RabbitMQ servers. Instead, your workers would be subscribed to three different queues – A, B and C (I would give them more meaningful names). Tasks that need to be executed on server(s) your refer to as "A" simply get sent to the A queue, and the same goes for all others. Celery worker running on server A is subscribed only to the queue A (celery ... worker -n node_A -Q A). Similar story is with workers running on servers B and C.
Once you have this kind of infrastructure in place you can then start refactoring the way you orchestrate those tasks. When you find yourself in need to call .get() in your task, you may immediately think that your are doing it wrong. The correct way is to use Celery Workflows, but it requires some kind of mental adaptation on our side. – I’ve learned this the hard way… Think of it this way, instead of imperatively saying what you need to be executed you should tell celery to run various tasks, and when you need to do something with results you collect them in another task, and so on. In practice most of your workflows will have a final task that collects results from all other tasks executed as part of that workflow, and do the final processing of results, so no .get() is called explicitly. This also makes it possible to easily distribute (and scale).
Here I’m answering to my own question,
In my case this answer didn’t work. So I read a bit more and I updated the code like below :
timeout = 0.1
while not ML_task.ready():
time.sleep(timeout)
result = ML_task.get(disable_sync_subtasks=False)
Hope it helps.
I want to break down a project to small microservices. I’ve been thinking of different tools like gRPC, but I think celery might be a better option for me. Please if I’m wrong and celery is not a good choice, tell me why.
What exactly I want to do?
- For example, I have three different servers, A, B and C. "A" must be able to send a request to "B". "B" must be listening all the time.
- "B" get request from "A", works on it, and then it must be able to send request to "C".
- "C" does some stuff with the given request from "B" and then it must return the result to "B".
- "B" works on the result and when it’s finished, it must return the result to "A".
What have I done?
- I created two RabbitMQ servers, one for "A" & "B" and another one for "B" and "C".
- I used send_task to send tasks to servers. One task to send a string from "A" to "B". "B" is already listening and waiting for requests from "A". There’s another send_task on "B" which sends a request to "C". "C" is also already running and waiting for requests from "B".
- I used backend=’rpc’ to return results.
When a request is sent by "A", "B" receives the request and send_task to "C". "C" also receives the request with no problem. The problem is returning results.
I tried different things, but I mostly got runtimeError: Never call result.get() within a task Celery error.
Does anyone have any solution?
This particular error indicates that using result.get() within a task is a bad practice that could lead to deadlocks.
If you know what you are doing you can try allow_join_result but that could cause performance issues:
with allow_join_result():
task.get()
You can also wait until the task is succeed:
timeout = 0.1
while not result.ready():
time.sleep(timeout)
ret = result.get()
I would rethink the whole thing if I were you.
First, I think there is no reason for multiple RabbitMQ servers. Instead, your workers would be subscribed to three different queues – A, B and C (I would give them more meaningful names). Tasks that need to be executed on server(s) your refer to as "A" simply get sent to the A queue, and the same goes for all others. Celery worker running on server A is subscribed only to the queue A (celery ... worker -n node_A -Q A). Similar story is with workers running on servers B and C.
Once you have this kind of infrastructure in place you can then start refactoring the way you orchestrate those tasks. When you find yourself in need to call .get() in your task, you may immediately think that your are doing it wrong. The correct way is to use Celery Workflows, but it requires some kind of mental adaptation on our side. – I’ve learned this the hard way… Think of it this way, instead of imperatively saying what you need to be executed you should tell celery to run various tasks, and when you need to do something with results you collect them in another task, and so on. In practice most of your workflows will have a final task that collects results from all other tasks executed as part of that workflow, and do the final processing of results, so no .get() is called explicitly. This also makes it possible to easily distribute (and scale).
Here I’m answering to my own question,
In my case this answer didn’t work. So I read a bit more and I updated the code like below :
timeout = 0.1
while not ML_task.ready():
time.sleep(timeout)
result = ML_task.get(disable_sync_subtasks=False)
Hope it helps.