ThreadPoolExecutor resubmit tasks when previous thread completed

Question

been through documents and google i could not find clear way to implement while True loop using ThreadPoolExecutor , what i need is submit task wait for it to be done and when its done reuse the thread and submit the new task and this should go on as long as sighup is received .
I came up with following but some how i feel there should be cleaner way to implement this

import time
from concurrent.futures import ThreadPoolExecutor, as_completed
import  requests
from threading import current_thread

my_wallet="somewallet"

def perform_fetch ():
        time.sleep(5)

        response_balance = requests.get("https://api.nanopool.org/v1/eth/balance/"+my_wallet)
        response_avg_hashrate= requests.get("https://api.nanopool.org/v1/eth/avghashrate/"+my_wallet)
        re_json_balance= response_balance.json()
        re_json_avg_hashrate=response_avg_hashrate.json()
        thread = current_thread()
        ret= "status is {} and blance is {:.8f} and avg hashrate is {:.2f} .thread name is {}"
            .format(re_json_balance['status'],re_json_balance['data'],re_json_avg_hashrate['data']['h24'],thread.name)
        return (ret)


with ThreadPoolExecutor(max_workers=2) as executor:

    futures = [executor.submit(perform_fetch) for i in range(2)]
    while True:
        time.sleep(1)
        for fut in futures :
            if(fut.running()):
                print("Task is running no result yet")
            elif(fut.done()):
                print(fut.result())
                futures.append(executor.submit(perform_fetch) )

    #for f in as_completed(futures):
    #    print(f.result())

I may also be able to implement the while True loop at function level (perform_fetch) and eliminate the need to resubmit the task, was wondering what is the best way for this as it seems to be very general usage of ThreadPoolExecutor.

Asked By: Amir

||

Source

Answer 1

In general, using a ThreadPoolExecutor like this seems like a pretty straghtforward way of solving the problem.

That only comment I would have is that you are modifying a list while iterating over it. It probably won’t crash since you are appending to the list rather than removing from it. But it is generally considered a bad idea.
Therefore I would suggest the following change.


    futures = [executor.submit(perform_fetch) for i in range(2)]
    while True:
        time.sleep(1)
        additions = []
        removals = []
        for fut in futures :
            if(fut.running()):
                print("Task is running no result yet")
            elif(fut.done()):
                print(fut.result())
                removals.append(fut)
                additions.append(executor.submit(perform_fetch))
        futures += additions
        for r in removals:
            futures.remove(r)
        del removals

As you say, you could also move a while loop into perform_fetch and keep the threads around forever.
But you would have to "return" the results by adding them to a global list. And access to that list would have to be protected by a Lock so you don’t have multiple threads trying to change it at the same time. That would also work, but I doubt it would be less complex. Make sure to use the lock in a with-statement, so you don’t have to balance acquire and release manually.

Answered By: Roland Smith

ThreadPoolExecutor resubmit tasks when previous thread completed

Question:

Answers: