Squish test functions always pass when used with squishtest module

Question

We are using Squish for Qt 6.6.2 on Windows 10 with Python 3.8.7 and running our tests using squishtest module with Robot Framework 4.0.1.

We are having an issue with the test functions provided by the Squish API where any verifications done with such a call (for example squishtest.test.imagePresent) will Pass. The issue itself was quite simple to pinpoint to the fact that although the verification failed, the function call itself was passing without raising exceptions. This can also be verified from the report provided by the squishrunner where we have <scriptedVerificationResult type="FAIL" time="--"> on the passed execution.

The question is, can we in any way get the actual verification result passed to the Robot so we can fail the test accordingly? Preferrably in real time rather than parsing the report afterwards.

In Squish this works perfectly fine

def main():
    startApplication("AUT")
    snooze(2)
    test.imagePresent("image.png", {"tolerant": True, "threshold": 85}, 
    waitForObjectExists(names.sceneContainer_GraphWidget))

but with Robot this is always passing

# In testSuite.robot

*** Settings ***
Library     MySquishLib

*** Test Cases ***
Test Image
    Start AUT
    Verify Image    image.png    {"tolerant": True, "threshold": 85}    names.sceneContainer_GraphWidget

# In MySquishLib.py

import squishtest
import names

def start_aut():
    squishtest.startApplication("AUT")

def verify_image(imageFile, imageParams, imageArea):
    squishtest.test.imagePresent(imageFile, imageParams, imageArea)

Asked By: Morkkis

||

Source

Answer 1

The test functions are not supposed to raise exceptions during execution in order to allow test to continue even if a single VP was failed. The function does however return a boolean value just as expected. By using

def verify_image(imageFile, imageParams, imageArea):
    if not squishtest.test.imagePresent(imageFile, imageParams, imageArea):
        raise Exception("Image was not found")

I’m able to fail the Robot test without any issues.

Answered By: Morkkis

Answer 2

Have a look at the documented: bool testSettings.throwOnFailure flag.
In your robot you can set this and you would not have to patch / rewrite every test.vp, test.compare, … method.

When running from within the Squish IDE, this flag can be unset. Probably robotframework provides some environment variables to detect wether the test case is running from within itself.

https://doc.froglogic.com/squish/latest/rgs-squish.html#rgs-testsettings

Answered By: Eugen_R

Squish test functions always pass when used with squishtest module

Question:

Answers: