How can I "fire and forget" a task without blocking main thread?

You could try something like this:

import multiprocessing


class MPPool:
    def __init__(self, num=multiprocessing.cpu_count() - 1):
        self.pool = multiprocessing.Pool(num)

    def __call__(self, f, *args, **kwargs):
        self.pool.apply_async(f, args=args, kwds=kwargs)


def run_and_forget(f, *args, **kwargs):
    if "pool" not in run_and_forget.__dict__:
        run_and_forget.pool = MPPool()

    run_and_forget.pool(f, *args, **kwargs)


if __name__ == '__main__':
    import time

    def test(n):
        time.sleep(n)
        print(f"done {n}")

    for i in range(20):
        run_and_forget(test, i)
        print(f"passed {i}")

    time.sleep(50)
    print("end")

The function run_and_forget can be used anywhere (within a single process), as the member pool is static-like and therefore defined at the first call.

This wasn’t fully tested, but I’ve provided some quick test code to see how it works. First things that comes to mind is that before exiting it would be smart to cleanup the multiprocessing pool.

I’ll answer "what you’ve asked", but I’ll preface that you may be asking the wrong question due to a lack of understanding.

In Python stdlib, subprocess can spin up separate independent processes that behave like "fire and forget". Here’s a couple:

import os, subprocess
subprocess.Popen(['mkdir', 'foo'])
os.popen('touch answer_is_$((1 + 2))')

It’d be much better to provide concrete examples of these "small and fast non-blocking tasks" you’d like to have, complete with the environment you’ll want them to be running in. You’re missing some understanding that’s evident b/c some of your statements conflict with others. For example, asyncio and threading don’t operate like "fire and forget" at all.

Also, there’s not going to be a good way to "background within any context" b/c the differences between different contexts matter, and "what’s best" depends on many factors.

To be able to "forget" you need to tie the task to an instance existence, so that it will init/del in reference to something you are already using, the example you added has nothing wrong but when is so short to implementation it is more a "fit that exact need" and not a general use case. You can extend that example to make stuff async can’t do as easily, in my example I went for a "mutable" variable, self won’t change during the instance lifetime (but the content can change without changing the address of self) so you can both "forget" of the task during the existence and feed it different values just because it takes self as an argument.

This async implementation is a very simplistic one, it consists of a list that gets added to the task you need and returns the "handle" to get the return or terminate it,
each time a thread pops a call it is getting removed or readded, this is to keep it as a FIFO list, already executed calls must be added as a new one and not an old one

import threading as th
import time

class CTask(object):

    once  = 1
    loop  = 8

    def __init__(self, foo, args, kwargs, type):
        self.__kwargs = kwargs
        self.__args = args
        self.__foo = foo
        self.__ret = None

        self.task_type = type
        self.terminated = False

    def pool_call(self):
        self.__ret = self.__foo(
            *self.__args,
            **self.__kwargs
        )
        if self.task_type == self.once:
            self.terminated = True

    def __call__(self):
        return self.__ret

    def terminate(self):
        self.terminated = True

class Worker_Pool:

    def push_task(self, type=CTask.once):
        def func_wraper(func):
            def varg_wraper(*a, **b):
                ntask = CTask(func, a, b, type)
                self.__lock.acquire(True, -1)
                self.__ref.append(ntask)
                self.__lock.release()
                return ntask
            return varg_wraper
        return func_wraper

    def __init__(self, n_work):
        self.__lock = th.Lock()
        self.__pool = []
        self.__ref = []
        self.alive = True

        for _ in range(n_work):
            self.__pool.append(
                th.Thread(target=self.__run)
            )
        for w in self.__pool:
            w.start()

    def __run(self):
        while self.alive:
            wtask = None
            if self.__lock.acquire(True, 0.1):
                if len(self.__ref) > 0:
                    wtask = self.__ref.pop(0)
                self.__lock.release()

            if wtask is None or wtask.terminated:
                continue

            wtask.pool_call()

            if wtask.task_type == CTask.loop:
                self.__lock.acquire(True, -1)
                self.__ref.append(wtask)
                self.__lock.release()

    def terminate(self):
        self.alive = False
        for w in self.__pool:
            w.join()

bg = Worker_Pool(3)

class Some_App:

    @bg.push_task(CTask.loop)
    def do_task(self):
        print(f'[{self.id}] -> {self.num}')
        time.sleep(0.5)

    @bg.push_task()
    def do_mult(self):
        self.num *= self.mul

    def __init__(self, id, imul):
        self.mul = imul
        self.num = 1
        self.id = id

        self.bg_task = self.do_task()

    def __del__(self):
        # you should make sure that the the 
        # function is no longer in execution
        # before continuing this function
        self.bg_task.terminate()

if __name__ == '__main__':
    inst_a = Some_App(1, 2)
    task_a = inst_a.do_task()
    inst_b = Some_App(5, 6)
    task_b = inst_b.do_task()

    print('')
    inst_a.do_mult()
    inst_b.do_mult()
    time.sleep(0.6)

    print('')
    inst_a.do_mult()
    inst_b.do_mult()
    time.sleep(1.2)

    bg.terminate()

Other answers use multiprocessing/subprocess and for a light task I would describe those as shooting yourself in the foot, subprocess uses text to communicate and the process uses sockets+picke so it is not the best choice.

The solutions you mentioned in your questions achieve what you ask "fire and forget" but the hidden question I hear is what’s the efficient / common way to do it.

Like many problems in computer science the answer is – it depends. I’ll try to explain.
Basically you have two methods: execute in a separate thread or a process.

Using threads you get shared memory access, and they’re lighter in terms of resource usage. It also performs context-switching better than process context-switching.

With processes, you can utilize more CPUs, but you lose the shared memory and (depends on the amount of processes you run vs. number of cores) you may run into more context-switching. (e.g, if you run a container with 2 cpus and 8 processes, they all gonna race for CPU time and more context-switch is probably to happen).

For a concrete example, let’s consider two scenarios. If your app needs to do CPU intensive tasks (e.g, encrypting, compressing, etc.) you would probably get better performance utilizing more CPUs.

On the other hand, if your tasks are blocking due to I/O (e.g, waiting for network, reading from disk, etc.) using threads probably be faster than using difference process for that.

Another thing to note is the communication between your TaskManager and the TaskExecutor. If the two needs to communicate your performance will be better using threads since the shared memory access.

To summarize, the better way to do it is to run performance tests because it really depends on the work you need to get done. My suggestion is start with the simpler solution (threads), get a baseline for your performance results and then try out processes. Running programs in multi-processing environment is considered more complex than multi-threaded.

My 2cent – when dealing with performance questions, you gotta have a baseline and compare to it, otherwise you’re guessing in the dark and spend your time tweaking the wrong parts of your application.