Best way to output concurrent task results in shell on the go

Question

That is maybe a very basic question, and I can think of solutions, but I wondered if there is a more elegant one I don’t know of (quick googling didn’t bring anything useful).

I wrote a script to communicate to a remote device. However, now that I have more than one of that type, I thought I just make the communication in concurrent futures and handle it simultaneously:

with concurrent.futures.ThreadPoolExecutor(20) as executor:
    executor.map(device_ctl, ids, repeat(args))

So it just calls up to 20 threads of device_ctl with respective IDs and the same args. device_ctl is now printing some results, but since they all run in parallel, it gets mixed up and looks messy. Ideally I could have 1 line per ID that shows the current state of the communication and gets updated once it changes, e.g.:

Dev1 Connecting...
Dev2 Connected! Status: Idle
Dev3 Connected! Status: Updating

However, I don’t really know how to solve it nicely. I can think of a status list that outside of the threads gets assembled into one status string, which gets frequently updated. But it feels like there could be a simpler method! Ideas?

Asked By: André

||

Source

Answer 1

Since there was no good answer, I made my own solution, which is quite compact but efficient. I define a class that I call globally. Each thread populates it or updates a value based on its ID. The ID is meant to be the same list entry as taken for the thread. Here I made a simple example how to use it:

class collect:
    ids = []
    outs = []
    LINE_UP = "33[1A"
    LINE_CLEAR = "x1b[2K"
    printed = 0

    def init(list):
        collect.ids = [i for i in list]
        collect.outs = ["" for i in list]
        collect.printall()

    def write(id, out):
        if id not in collect.ids:
            collect.ids.append(id)
            collect.outs.append(out)
        else:
            collect.outs[collect.ids.index(id)] = out

    def writeout(id, out):
        if id not in collect.ids:
            collect.ids.append(str(id))
            collect.outs.append(str(out))
        else:
            collect.outs[collect.ids.index(id)] = str(out)
        collect.printall()

    def append(id, out):
        if id not in collect.ids:
            collect.ids.append(str(id))
            collect.outs.append(str(out))
        else:
            collect.outs[collect.ids.index(id)] += str(out)

    def appendout(id, out):
        if id not in collect.ids:
            collect.ids.append(id)
            collect.outs.append(out)
        else:
            collect.outs[collect.ids.index(id)] += str(out)
        collect.printall()

    def read(id):
        return collect.outs[collect.ids.index(str(id))]

    def readall():
        return collect.outs, "n".join(collect.outs)

    def printall(filter=""):
        if collect.printed > 0:
            print(collect.LINE_CLEAR + collect.LINE_UP * len(collect.ids), end="")
        print(
            "n".join(
                [
                    collect.ids[i] + "t" + collect.outs[i] + " " * 30
                    for i in range(len(collect.outs))
                    if filter in collect.ids[i]
                ]
            )
        )

        collect.printed = len(collect.ids)

def device_ctl(id,args):
    collect.writeout(id,"Connecting...")
    if args.connected:
        collect.writeout(id,"Connected")

collect.init(ids)

with concurrent.futures.ThreadPoolExecutor(20) as executor:
    executor.map(device_ctl, ids, repeat(args))

Answered By: André

Best way to output concurrent task results in shell on the go

Question:

Answers: