run functions in paralel Python

Question

I have a stream and i have a function I want to run, when i receive the message on this stream

  async def some_func():
    asyncio.sleep(5)
    print("hello world")

  client = create_client('wax.dfuse.eosnation.io:9000')
  stream = client.Execute(Request(query = OPERATION_EOS))

  for rawRequest in stream:
    async.gather(some_func())

If there are 2 or more messages at the same time I want 2 or more functions that run in parallel.

Currently this script does not run a function

I need just a way to run function independently from main function.

Asked By: Александр Третинов

||

Source

Answer 1

with threading you could do

import threading

thread1 = threading.Thread(target=fn)
thread2 = threading.Thread(target=fn)

thread1.start()
thread2.start()

thread1.join()
thread2.join()

Answered By: Anthony L

Answer 2

Code example:

import asyncio
import time
chain = ""
sum = 0

async def myproc(callid):
    global chain
    global sum
    print(f"myProc {callid} started ...")
    t1 = time.perf_counter()
    time.sleep(2.5)
    chain = chain + "->" + str(callid)
    sum = sum + 1
    await asyncio.sleep(5)
    print("hello world")
    t = time.perf_counter() - t1
    print(f"   myProc {callid} finished in {t:0.5f} seconds. sum = {sum} chain {chain}")

async def main():
    #client = create_client('wax.dfuse.eosnation.io:9000')
    #stream = client.Execute(Request(query = OPERATION_EOS))
    stream = range(10) # # simulation of the task aka each eelment from stream
    coros = [myproc(rawRequest) for rawRequest in stream]    
    await asyncio.gather(*coros)
    
if __name__ == "__main__":
    start_sec = time.perf_counter()
    await main() # # for notebook how does work, for python interpreter use asyncio.run(main())
    
    elapsed_secs = time.perf_counter() - start_sec
    print(f"Job finished in {elapsed_secs:0.5f} seconds.")

Output:

myProc 0 started ...
myProc 1 started ...
myProc 2 started ...
myProc 3 started ...
myProc 4 started ...
myProc 5 started ...
myProc 6 started ...
myProc 7 started ...
myProc 8 started ...
myProc 9 started ...
hello world
   myProc 0 finished in 25.02580 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 1 finished in 22.52303 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 2 finished in 20.02011 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 3 finished in 17.51737 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 4 finished in 15.01457 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 5 finished in 12.51187 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 6 finished in 10.00907 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 7 finished in 7.50854 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 8 finished in 7.50605 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
hello world
   myProc 9 finished in 7.50515 seconds. sum = 10 chain ->0->1->2->3->4->5->6->7->8->9
Job finished in 30.02882 seconds.

For a detailed explanation u can look at following very good explanation on asynchronous execution of individual functions inside a program instead of parallelizing the processing, so instead of parallel execution using threading that is not efficient u can leverage a combination and the asyncio.gather(*coros) does run it all in parallel format without defining threads and increasing infrastructure Async Processing in Python – Make Data Pipelines Scream. Consider also using the asyncio.run() function instead of using lower level functions to manually create and close an event-loop, I did point out in comment but would be additional loop and for me in notebook that already does run by default so a high-level API for coroutines run is better in this case to handle all and u need to execute those coroutines with "event-loop" following format asyncio.run(main()) instead a simple call for coroutines main() (a bit of non-sense explanation for better understanding all this APIs).

Note: I did use notebook to execute it so if u do use python interpreter then use asyncio.run(main()) instead of await main() We use async processing here to mimic parallel processing, instead of doing true parallel processing which is generally harder to accomplish and not suited for your streaming job.

Answered By: n1tk

Answer 3

Somehow I made it working.

My code:

async def some_func(rawResult):
  # There is some code

async def stream_eosio(loop):
  for rawResult in stream:
    asyncio.run_coroutine_threadsafe(some_func(rawResult), loop)

if __name__ == "__main__":
  loop = asyncio.new_event_loop()
  Thread(target=asyncio.run, args=(stream_eosio(loop),)).start()
  loop.run_forever()

Cons:
You can’t stop this script with Ctrl + Z or Ctrl + C, because of Thread.

Pros: It’s kinda ez.

Answered By: Александр Третинов

run functions in paralel Python

Question:

Answers: