Set niceness of each process in a multiprocessing.Pool

Question

How can I set the niceness for each process in a multiprocessing.Pool? I understand that I can increment niceness with os.nice(), but how do call it in the child process after creating the pool? If I call it in the mapped function it will be called every time the function executes, rather than once when the process is forked.

import multiprocessing as mp    

NICENESS = 19
DATA = range(100000)

def foo(bar):
    return bar * 2

pool = mp.Pool(100)
# Somehow set niceness of each process to NICENESS

pool.map(foo, DATA)

Asked By: jsj

||

Source

Answer 1

You can access the worker processes via pool._pool.
With this you can probably set the nicesness of each worker individually.

import time
import psutil
import multiprocessing as mp
NICENESS =19
DATA = range(15)

def foo(bar):
    time.sleep(bar)
    return bar*2
if __name__=='__main__':
    pool = mp.Pool(8) # 100 might not make sense if you only have 8 cores

    processes = [p.pid for p in pool._pool]
    for pid in processes:
        p = psutil.Process(pid)
        p.nice(NICENESS) # POSIX
        # use this for Windows: 
        # p.nice(psutil.HIGH_PRIORITY_CLASS)
    pool.map(foo, DATA)

I cannot test it on Linux, as I’m on Windows, but here it works well. Let me know if it works on Linux. Might be that you need to run the parent process as sudo, as there’s something with not being able to elevate other processes.

Answered By: skjerns

Answer 2

What about using an initializer for that? https://docs.python.org/3.8/library/multiprocessing.html#multiprocessing.pool.Pool
The function is called once when the pool is started so the os.nice() call in the initializer sets the niceness for the proces after that.

I’ve added some additional statements to show that it works in your worker function but the os.nice() calls should obviously be removed since you want a static niceness value.

import multiprocessing as mp
import os

NICENESS = 3
DATA = range(6)


def foo(bar):
    newniceness = os.nice(1) # remove this
    print('Additional niceness:', newniceness) # remove this
    return bar * 2


def set_nicesness(val): # the initializer
    newval = os.nice(val) # starts at 0 and returns newvalue
    print('niceness value:', newval)



pool = mp.Pool(3, initializer=set_nicesness, initargs=(NICENESS,))
# Somehow set niceness of each process to NICENESS
pool.map(foo, DATA)

As you can see from the prints the niceness now starts at 3 (I’ve set this for NICENESS) and starts incrementing from there.

Or as a useable snippet

import multiprocessing as mp
import os

NICENESS = 3


def mp_function(bar: int) -> int:
    return bar * 2


if __name__ == '__main__':
    pool = mp.Pool(3, initializer=os.nice, initargs=(NICENESS,))
    data = range(6)
    pool.map(mp_function, data)

Answered By: Marc

Set niceness of each process in a multiprocessing.Pool

Question:

Answers: