Python Ram usage increases after new thread starts

Question

In a True while loop, I’m starting a new Thread which in turn starts a shell Subprocess (python script). Due to other reasons, the python scripts need to be called using the shell subprocess. The subprocess runs for a certain time and then quits.

After each thread is created the main program (cmd) requires more RAM usage.

Code for starting thread:

while True:
  cameras = camera_queue.get_next_cameras()
  session_id = uuid.uuid4()

  task = threading.Thread(target=start_thread, args=(cameras, f'{session_id}'))
  task.start()
  time.sleep(camera_queue.batch_size * CAMERA_INTERVAL)

Subprocess code (start_thread):

subprocess.Popen(
  f'start /MIN /wait python Backend\Recorder\secondary.py "{cameras}" "{logger_id}"',
  shell=True,
)

This program will start around 1000 subprocesses per hour. I have checked and the subprocesses do close and there never exists more subprocesses than 1000.

Is there a way to clean up the usage? If you force quit the current cmd usage the ram usage instantly disappears while the shells stay open. After running the program for 1 hour the ram usage is at 3GB and after around 10 hours around 15-20GB.

Asked By: Casper Kristiansson

||

Source

Answer 1

You can get rid of the threads, shell and start.

First, create Popen objects directly from the main thread.

Second, when creating subprocesses on ms-windows, there are ways to prevent/hide their windows. Since I’m not an ms-windows user, I’ve pulled this from the documentation;

The first thing I would try is to specify creationflags=subprocess.CREATE_NO_WINDOW to prevent the process from opening a window. Note that you can use | to combine creationflags.
If that doesn’t work, create a subprocess.STARTUPINFO(dwFlags=subprocess.wShowWindow, wShowWindow=subprocess.SW_HIDE) and pass that to the Popen constructor.

Third, when that is done, you don’t need start anymore, so you can also get rid of shell=True.

proc = subprocess.Popen(
    ["python", 
     "Backend\Recorder\secondary.py",
     f"{cameras}",
     f"{logger_id}"],
    creationflags=subprocess.CREATE_NO_WINDOW
)

This link points to a script that manages a herd of subprocesses. In this case they are running the convert program from ImageMagick, but the approach is general.

Answered By: Roland Smith

Answer 2

@RolandSmith solution half worked. For some reason, by just passing the creation flag the scripts would never run / would instantly give a popen exception. A solution to this was to pass the shell=True still.

subprocess.Popen(
  ["python",
    "Backend\Recorder\dcr_unit.py",
    f"{cameras}",
    f"{session_id}"],
  creationflags=subprocess.CREATE_NO_WINDOW,
  shell=True
)

I noticed that for some reason the popen objects would still run in the background even though the instance was done executing its code. But when passing the creation flags and then running the script just through cmd the memory leak was pretty much 0. For some reason when running this script through a code editor like vscode would cause the memory leak tenfold. The program accumulated around 10GB of ram per 10000 new instances.

Answered By: Casper Kristiansson

Python Ram usage increases after new thread starts

Question:

Answers: