Why does a result() from concurrent.futures.as_completed() occasionally return None?

Question

I’ve been writing a fairly complex series of modules to help test our poorly documented networking gear, this one focused on trying the various passwords used across the company. I’ve been using the concurrent.futures module to speed things along by testing many devices in parallel but am running in to a problem where occasionally a result comes back as None and it’s a mystery to me as why. It revolves around this chunk of code:

def process_device(dev):
    check = CheckPass(creds, inv_list)
    dev = switch.Switch(**dev)
    online, net_device = check.device_online(dev)
    if online == True:
        conn, net_device = check.try_passwords(dev, return_conn=True)
        if conn != False and conn != "none":
            if conn != True:
                hostname,model,serial_num,version,date = net_device.get_id_details(conn)
                net_device.hostname = hostname
                net_device.model = model
                net_device.serial_num = serial_num
                net_device.version = version
                net_device.date = date
                conn.disconnect()
            conf_dev = check.confirm_device(net_device)
            check.dev_confirmed.append(conf_dev)
            return dev.hostname, dev.ip        

with concurrent.futures.ThreadPoolExecutor(max_workers=120) as executor:
    threads = {executor.submit(process_device, dev): dev for dev in inv_list}
    for future in concurrent.futures.as_completed(threads):
        name, ip = future.result()

At unpredictable intervals future.result() will have a result of None and I can’t reliably reproduce it. The error occurs with different devices every time, I’ve checked the logs and fed it an inventory list containing just the device that it had processed and running that device by itself will be fine. I have tried using fewer workers and more. Clearly I’m not understanding something about how future.result() works. dev.hostname and dev.ip always have values so process_device() should always return them (barring unhandled exceptions, which haven’t occurred) yet I always end up with TypeError: Cannot unpack non-iterable NoneType object referencing the name, ip = future.results() command.

Asked By: claccx

||

Source

Answer 1

It is not problem with future but with your function process_device() which sometimes can return None.

When online is False or when if conn != False and conn != "none": gives False then process_device() will run default return None

def process_device(dev):

    # ... your code ...

    return None  # default behavior

You should filter results

result = future.result()
if result:
    name, ip = result

Answered By: furas

Why does a result() from concurrent.futures.as_completed() occasionally return None?

Question:

Answers: