Run multiple python scripts concurrently
Question:
How can I run multiple python scripts? At the moment I run one like so python script1.py.
I’ve tried python script1.py script2.py and that doesn’t work: only the first script is run. Also, I’ve tried using a single file like this;
import script1
import script2
python script1.py
python script2.py
However this doesn’t work either.
Answers:
The simplest solution to run two Python processes concurrently is to run them from a bash file, and tell each process to go into the background with the & shell operator.
python script1.py &
python script2.py &
For a more controlled way to run many processes in parallel, look into the Supervisor project, or use the multiprocessing module to orchestrate from inside Python.
With Bash:
python script1.py &
python script2.py &
That’s the entire script. It will run the two Python scripts at the same time.
Python could do the same thing itself but it would take a lot more typing and is a bad choice for the problem at hand.
I think it’s possible though that you are taking the wrong approach to solving your problem, and I’d like to hear what you’re getting at.
You try the following ways to run the multiple python scripts:
import os
print "Starting script1"
os.system("python script1.py arg1 arg2 arg3")
print "script1 ended"
print "Starting script2"
os.system("python script2.py arg1 arg2 arg3")
print "script2 ended"
Note: The execution of multiple scripts depends purely underlined operating system, and it won’t be concurrent, I was new comer in Python when I answered it.
Update: I found a package: https://pypi.org/project/schedule/
Above package can be used to run multiple scripts and function, please check this and maybe on weekend will provide some example too.
i.e:
import schedule
import time
import script1, script2
def job():
print("I'm working...")
schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every(5).to(10).days.do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)
while True:
schedule.run_pending()
time.sleep(1)
I am working in Windows 7 with Python IDLE.
I have two programs,
# progA
while True:
m = input('progA is running ')
print (m)
and
# progB
while True:
m = input('progB is running ')
print (m)
I open up IDLE and then open file progA.py. I run the program, and when prompted for input I enter "b" + <Enter> and then "c" + <Enter>
I am looking at this window:
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
= RESTART: C:UsersMikeAppDataLocalProgramsPythonPython36-32progA.py =
progA is running b
b
progA is running c
c
progA is running
Next, I go back to Windows Start and open up IDLE again, this time opening file progB.py. I run the program, and when prompted for input I enter "x" + <Enter> and then "y" + <Enter>
I am looking at this window:
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
= RESTART: C:UsersMikeAppDataLocalProgramsPythonPython36-32progB.py =
progB is running x
x
progB is running y
y
progB is running
Now two IDLE Python 3.6.3 Shell programs are running at the same time, one shell running progA while the other one is running progB.
You can use Gnu-Parallel to run commands concurrently, works on Windows, Linux/Unix.
parallel ::: "python script1.py" "python script2.py"
I do this in node.js (on Windows 10) by opening 2 separate cmd instances and running each program in each instance.
This has the advantage that writing to the console is easily visible for each script.
I see that in python can do the same: 2 shells.
You can run multiple instances of IDLE/Python shell at the same time. So open IDLE and run the server code and then open up IDLE again, which will start a separate instance and then run your client code.
I had to do this and used subprocess.
import subprocess
subprocess.run("python3 script1.py & python3 script2.py", shell=True)
The most simple way in my opinion would be to use the PyCharm IDE and install the ‘multirun’ plugin. I tried alot of the solutions here but this one worked for me in the end!
Adding a wait at the end of the bash script would be advisable, as the script exits when one of the process completes. If we need the script to exit only after all the python process are completed, add a wait at the end of the bash script.
So the script would be
#!/bin/bash
python script1.py ;
python script2.py &
python script3.py &
wait
; at the end of first script is used to run script1 & once finished then start with script2 & script3 in parallel and wait till all of the 3 scripts are completed.
you can use
import os
os.system("script1.py && script2.py")
and so on you can add ou as many scripts as you want: just separate with &&
This is for Windows OS
How can I run multiple python scripts? At the moment I run one like so python script1.py.
I’ve tried python script1.py script2.py and that doesn’t work: only the first script is run. Also, I’ve tried using a single file like this;
import script1
import script2
python script1.py
python script2.py
However this doesn’t work either.
The simplest solution to run two Python processes concurrently is to run them from a bash file, and tell each process to go into the background with the & shell operator.
python script1.py &
python script2.py &
For a more controlled way to run many processes in parallel, look into the Supervisor project, or use the multiprocessing module to orchestrate from inside Python.
With Bash:
python script1.py &
python script2.py &
That’s the entire script. It will run the two Python scripts at the same time.
Python could do the same thing itself but it would take a lot more typing and is a bad choice for the problem at hand.
I think it’s possible though that you are taking the wrong approach to solving your problem, and I’d like to hear what you’re getting at.
You try the following ways to run the multiple python scripts:
import os
print "Starting script1"
os.system("python script1.py arg1 arg2 arg3")
print "script1 ended"
print "Starting script2"
os.system("python script2.py arg1 arg2 arg3")
print "script2 ended"
Note: The execution of multiple scripts depends purely underlined operating system, and it won’t be concurrent, I was new comer in Python when I answered it.
Update: I found a package: https://pypi.org/project/schedule/
Above package can be used to run multiple scripts and function, please check this and maybe on weekend will provide some example too.
i.e:
import schedule
import time
import script1, script2
def job():
print("I'm working...")
schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every(5).to(10).days.do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)
while True:
schedule.run_pending()
time.sleep(1)
I am working in Windows 7 with Python IDLE.
I have two programs,
# progA
while True:
m = input('progA is running ')
print (m)
and
# progB
while True:
m = input('progB is running ')
print (m)
I open up IDLE and then open file progA.py. I run the program, and when prompted for input I enter "b" + <Enter> and then "c" + <Enter>
I am looking at this window:
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
= RESTART: C:UsersMikeAppDataLocalProgramsPythonPython36-32progA.py =
progA is running b
b
progA is running c
c
progA is running
Next, I go back to Windows Start and open up IDLE again, this time opening file progB.py. I run the program, and when prompted for input I enter "x" + <Enter> and then "y" + <Enter>
I am looking at this window:
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 17:26:49) [MSC v.1900 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>>
= RESTART: C:UsersMikeAppDataLocalProgramsPythonPython36-32progB.py =
progB is running x
x
progB is running y
y
progB is running
Now two IDLE Python 3.6.3 Shell programs are running at the same time, one shell running progA while the other one is running progB.
You can use Gnu-Parallel to run commands concurrently, works on Windows, Linux/Unix.
parallel ::: "python script1.py" "python script2.py"
I do this in node.js (on Windows 10) by opening 2 separate cmd instances and running each program in each instance.
This has the advantage that writing to the console is easily visible for each script.
I see that in python can do the same: 2 shells.
You can run multiple instances of IDLE/Python shell at the same time. So open IDLE and run the server code and then open up IDLE again, which will start a separate instance and then run your client code.
I had to do this and used subprocess.
import subprocess
subprocess.run("python3 script1.py & python3 script2.py", shell=True)
The most simple way in my opinion would be to use the PyCharm IDE and install the ‘multirun’ plugin. I tried alot of the solutions here but this one worked for me in the end!
Adding a wait at the end of the bash script would be advisable, as the script exits when one of the process completes. If we need the script to exit only after all the python process are completed, add a wait at the end of the bash script.
So the script would be
#!/bin/bash
python script1.py ;
python script2.py &
python script3.py &
wait
; at the end of first script is used to run script1 & once finished then start with script2 & script3 in parallel and wait till all of the 3 scripts are completed.
you can use
import os
os.system("script1.py && script2.py")
and so on you can add ou as many scripts as you want: just separate with &&
This is for Windows OS