Python imap to take multiple functions

Question

I am utilizing multiprocessing using imap, since I need the results in order of the tasks/jobs executed. I just realized that I need a different function for the first task/job. Is there a way to specify 2 functions for the imap (where the first function (say func1) is only applied to the first task/job, and the rest will execute the second function (say func2)?

What I currently have is like:

from multiprocessing import Pool

pools = 2

with Pool(pools) as p:
    for result in p.imap(func2, args): # where args is an iterator/list of params to be passed to func2
        # some more processing here on the result

Which basically before I realized what I needed to do.

Is there a way to do what I am looking for with imap? Or would I just have to execute func1 first, then do the imap with the second function. That would work, but I really wanted the first task and the second task to run in parallel (when pools is set to 2 at least).

Assume there are hundreds of tasks/jobs to be performed.

Asked By: user1179317

||

Source

Answer 1

If I understand you correctly you can use itertools.chain:

from time import sleep
from itertools import chain
from multiprocessing import Pool


def func1(args):
    print("func1", args)
    sleep(1)


def func2(args):
    print("func2", args)
    sleep(2)


if __name__ == "__main__":
    pools = 2

    args = [1, 2, 3]

    with Pool(pools) as p:
        for result in chain(p.imap(func1, [args[0]]), p.imap(func2, args[1:])):
            pass

Prints (the first two tasks are executed in parallel):

func1 1  # in parallel with func2
func2 2  # in parallel with func1
func2 3

Answered By: Andrej Kesely

Python imap to take multiple functions

Question:

Answers: