How to run celery on windows?
Question:
How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker
?
Answers:
It’s done the same way as in Linux. Changing directory to module containing celery task and calling "c:pythonpython" -m celery -A module.celery worker
worked well.
yes:
celery -A your_application -l info
also note Celery have dropped support for Windows(since v4), so best to
pip install celery==3.1.25
3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn’t need to downgrade flower(browser monitor for celery) though.
See also the FAQ for Windows
Celery 4.0+
does not officially support window already. But it still works on window for some development/test purpose.
Use eventlet
instead as below:
pip install eventlet
celery -A <module> worker -l info -P eventlet
It works for me on window 10
+ celery 4.1
+ python 3
.
This solution solved the following exception:
[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
File "c:userswchen8workvenvweinstalibsite-packagesbilliardpool.py", line 358, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "c:userswchen8workvenvweinstalibsite-packagesceleryapptrace.py", line 525, in _fast_trace_task
tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack
===== update 2018-11 =====
Eventlet has an issue on subprocess.CalledProcessError:
https://github.com/celery/celery/issues/4063
https://github.com/eventlet/eventlet/issues/357
https://github.com/eventlet/eventlet/issues/413
So try gevent
instead.
pip install gevent
celery -A <module> worker -l info -P gevent
This works for me on window 10
+ celery 4.2
+ python 3.6
There are two workarounds to make Celery 4 work on Windows:
- use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
- set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)
See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details
I’ve made a .bat file besides my manage.py file with this code:
title CeleryTask
::See the title at the top.
cd
cmd /k celery -A MainProject worker -l info
so each time I want to run celery, I just double-click this batch file and it runs perfectly.
And the fact that you can’t use celery 4 on windows is true.
I have run celery task using RabbitMQ server.
RabbitMq is better and simple than redis broker
while running celery use this command “celery -A project-name worker –pool=solo -l info”
and avoid this command “celery -A project-name worker –loglevel info”
You can still use celery 4 0+ with Windows 10+
Just use this command “celery -A projet worker – -pool=solo – l info” instead of “celery – A project worker -l info
Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose.
you can use any of this:
celery worker --app=app.app --pool=eventlet --loglevel=INFO
celery worker --app=app.app --pool=gevent --loglevel=INFO
celery worker --app=app.app --pool=solo --loglevel=INFO
or using other format:
celery -A <app> worker --loglevel=info -P eventlet
celery -A <app> worker --loglevel=info -P gevent
celery -A <app> worker --loglevel=info -P solo
if you have get context
error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo
https://www.distributedpython.com/2018/08/21/celery-4-windows/
After feeling like killing myself by using celery 4.4 on windows, I think I can answer this question.
For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
Then run celery worker command with default pool option.
celery worker -A <celery_file> -l info
This will run celery worker concurrently with multiple child processes.
Note: When you run celery with gevent or eventlet pool, it will work but it won’t run concurrent processes on windows.
Compile Celery with –pool=solo argument.
Example:
celery -A your-application worker -l info --pool=solo
To run celery on windows for development. you have two methods.
- Use Windows as Host Machine.
- Install Redis Server from https://github.com/dmajkic/redis/downloads
- Run redis_server.exe
- Install Celery using
Pip install Celery
.
- Run Celery commands.
Limitations:
- This method is only supported for Python 3.6 or lower.
- Celery 4 does not provide support for windows.
- will be using Old Redis-server Version.
- Hard to manage
4. Using WSL for running Celery and Redis-server
- Install WSL using this guide https://www.windowscentral.com/install-windows-subsystem-linux-windows-10.
- Install redis-server using
sudo apt install redis-server
on the WSL terminal.
- Install Celery and Execute all celery-related commands on WSL.
WSL would be a friend for you on windows if you have already familiar with Linux.
With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.
You can run celery on windows without an extra library by using threads
celery -A your_application worker -P threads
Celery is testing it with windows in it’s CI again from version 5.x and you can try using it with windows if any issue occurs try set following environment variable in python code before creation of celery instance
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
But as far as I know celery now officially support running on windows for development purpose.
How to run celery worker on Windows without creating Windows Service? Is there any analogy to $ celery -A your_application worker
?
It’s done the same way as in Linux. Changing directory to module containing celery task and calling "c:pythonpython" -m celery -A module.celery worker
worked well.
yes:
celery -A your_application -l info
also note Celery have dropped support for Windows(since v4), so best to
pip install celery==3.1.25
3.1.25 was the last version that works on windows(just tested on my win10 machine). Didn’t need to downgrade flower(browser monitor for celery) though.
See also the FAQ for Windows
Celery 4.0+
does not officially support window already. But it still works on window for some development/test purpose.
Use eventlet
instead as below:
pip install eventlet
celery -A <module> worker -l info -P eventlet
It works for me on window 10
+ celery 4.1
+ python 3
.
This solution solved the following exception:
[2017-11-16 21:19:46,938: ERROR/MainProcess] Task handler raised error: ValueError('need more than 0 values to unpack',)
Traceback (most recent call last):
File "c:userswchen8workvenvweinstalibsite-packagesbilliardpool.py", line 358, in workloop
result = (True, prepare_result(fun(*args, **kwargs)))
File "c:userswchen8workvenvweinstalibsite-packagesceleryapptrace.py", line 525, in _fast_trace_task
tasks, accept, hostname = _loc
ValueError: need more than 0 values to unpack
===== update 2018-11 =====
Eventlet has an issue on subprocess.CalledProcessError:
https://github.com/celery/celery/issues/4063
https://github.com/eventlet/eventlet/issues/357
https://github.com/eventlet/eventlet/issues/413
So try gevent
instead.
pip install gevent
celery -A <module> worker -l info -P gevent
This works for me on window 10
+ celery 4.2
+ python 3.6
There are two workarounds to make Celery 4 work on Windows:
- use eventlet, gevent or solo concurrency pool (if your tasks as I/O and not CPU-bound)
- set the environment variable FORKED_BY_MULTIPROCESSING=1 (this is what actually causes the underlying billiard package to to fail under Windows since version 4)
See https://www.distributedpython.com/2018/08/21/celery-4-windows for more details
I’ve made a .bat file besides my manage.py file with this code:
title CeleryTask
::See the title at the top.
cd
cmd /k celery -A MainProject worker -l info
so each time I want to run celery, I just double-click this batch file and it runs perfectly.
And the fact that you can’t use celery 4 on windows is true.
I have run celery task using RabbitMQ server.
RabbitMq is better and simple than redis broker
while running celery use this command “celery -A project-name worker –pool=solo -l info”
and avoid this command “celery -A project-name worker –loglevel info”
You can still use celery 4 0+ with Windows 10+
Just use this command “celery -A projet worker – -pool=solo – l info” instead of “celery – A project worker -l info
Celery 4.0+ does not officially support window already. But it still works on window for some development/test purpose.
you can use any of this:
celery worker --app=app.app --pool=eventlet --loglevel=INFO
celery worker --app=app.app --pool=gevent --loglevel=INFO
celery worker --app=app.app --pool=solo --loglevel=INFO
or using other format:
celery -A <app> worker --loglevel=info -P eventlet
celery -A <app> worker --loglevel=info -P gevent
celery -A <app> worker --loglevel=info -P solo
if you have get context
error then upgrade gevent 20.6.2 and eventlet to 0.26.1 or use solo
https://www.distributedpython.com/2018/08/21/celery-4-windows/
After feeling like killing myself by using celery 4.4 on windows, I think I can answer this question.
For celery version 4.0 and above, first set following environment variable in python code before creation of celery instance.
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
Then run celery worker command with default pool option.
celery worker -A <celery_file> -l info
This will run celery worker concurrently with multiple child processes.
Note: When you run celery with gevent or eventlet pool, it will work but it won’t run concurrent processes on windows.
Compile Celery with –pool=solo argument.
Example:
celery -A your-application worker -l info --pool=solo
To run celery on windows for development. you have two methods.
- Use Windows as Host Machine.
- Install Redis Server from https://github.com/dmajkic/redis/downloads
- Run redis_server.exe
- Install Celery using
Pip install Celery
. - Run Celery commands.
Limitations:
- This method is only supported for Python 3.6 or lower.
- Celery 4 does not provide support for windows.
- will be using Old Redis-server Version.
- Hard to manage
4. Using WSL for running Celery and Redis-server
- Install WSL using this guide https://www.windowscentral.com/install-windows-subsystem-linux-windows-10.
- Install redis-server using
sudo apt install redis-server
on the WSL terminal. - Install Celery and Execute all celery-related commands on WSL.
WSL would be a friend for you on windows if you have already familiar with Linux.
With few limitations including support of Docker on WSL. you can perform most of the Development tasks using WSL.
You can run celery on windows without an extra library by using threads
celery -A your_application worker -P threads
Celery is testing it with windows in it’s CI again from version 5.x and you can try using it with windows if any issue occurs try set following environment variable in python code before creation of celery instance
os.environ.setdefault('FORKED_BY_MULTIPROCESSING', '1')
But as far as I know celery now officially support running on windows for development purpose.