How can I make this while True loop run faster and work properly? It detects the presence of 3 unwanted items on screen, when gone, alert triggered

Question:

I have a while True loop searching for the presence three different images. If any of the three are detected, ‘space’ is pressed, to search for another application. If an application is found without any of the three images, the loop is broken, and an alert is triggered notifying the user that there’s work to be done.

The problem I’m having is, I think pyautogui detects that the image is gone immediately after enter is pressed, and the next application is loading in my browser, as it loads, nothing is there, it’s a blank page. It only takes a fraction of a second to load the next application. When the next one loads in, often times, 1 of the 3 images are present but the loop already broke prematurely.

Here’s the loop:

while True:
    pyautogui.press("space")
    img = pyautogui.locateOnScreen("ffd1.PNG", confidence=0.7)
    img = pyautogui.locateOnScreen("ffd2.PNG", confidence=0.7)
    img = pyautogui.locateOnScreen("ffd3.PNG", confidence=0.7)
    if img == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break

Is there a better way to do this? I want it to press ‘space’ quickly, and search for the lack of my images at the same time. I tried adding a time.sleep(.5) in there, and it didn’t help at all. The loop still breaks between loading screens. 😐

somebody pointed out that I’m only searching for one image. I fixed that here… It’s still very slow.

while True:
    img = pyautogui.locateOnScreen("ffd1.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    img2 = pyautogui.locateOnScreen("ffd2.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    img3 = pyautogui.locateOnScreen("ffd3.PNG", confidence=0.7)
    if img == None and img2 == None and img3 == None:
        ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)
        break
    pyautogui.press("space")
    

tried rewriting using opencv and the thing just crashes all the time for no reason

img = cv2.imread('ffd1.PNG')
image_detected = False

while True:
    screenshot = pyautogui.screenshot()
    screenshot = np.array(screenshot)
    screenshot = cv2.cvtColor(screenshot, cv2.COLOR_RGB2BGR)
    
    res = cv2.matchTemplate(screenshot, img, cv2.TM_CCOEFF_NORMED)
    threshold = 0.8
    loc = np.where(res >= threshold)
    
    if np.any(loc) and not image_detected:
        print("Image detected on the screen! Pressed space key.")
        image_detected = True
        pyautogui.press('space')
        time.sleep(1)
        
    elif not np.any(loc) and image_detected:
        print("Image is no longer detected on the screen. Exiting...")
        break
Asked By: bigbird342d

||

Answers:

pyautogui is notoriously slow at screen capture and openCV and mss is/are alternatives to explore. That said, you are still often looping several times to set all values for img in order to do the test. Your opencv code is the closest to what I might expect.

For just pyautogui, try this:

while any(map(lambda f: pyautogui.locateOnScreen(f, confidence=0.7), ["ffd1.PNG", "ffd2.PNG", "ffd3.PNG"])):
    print("Image detected on the screen! Pressed space key.")
    pyautogui.press('space')
    time.sleep(1)

ctypes.windll.user32.MessageBoxW(0, "There's work to be done!!", "There's work to be done!!", 0x1000)

I’m not anticipating a huge improvement particularly if you have a larger monitor.

Answered By: JonSG
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.