Should pytest be used for integration testing an embedded system?

Question

I’m working on setting up my team’s new unit test and integration test infrastructure and want to make sure I’m starting off by selecting the correct test frameworks. I’m an embedded developer testing code running on a VxWorks operating system with a C/C++ production codebase.

We need a framework capable of directly testing C/C++ for unit testing, so for our unit tests I chose Googletest as our framework.

However, for integration tests we’ve generally tested using Python scripts (with no test framework). The Python scripts connect to the embedded system over a network and test cases via sending commands and receiving telemetry.

Would using pytest as a test framework be beneficial to the way we’re currently using Python for integration testing an embedded system? Most of the examples I’ve seen use pytest in a more unit test fashion by creating assertions for single functions in a Python production codebase.

EDIT:
Per hoefling’s comment, i’ll provide a (very simplified) example of one of our existing Python integration test cases, and also what I believe its corresponding Pytest implementation would be.

#Current example
def test_command_counter():
    preTestCmdCount = getCmdCountFromSystem()
    sendCommandToSystem()
    postTestCmdCount = getCmdCountFromSystem()

    if (postTestCmdCount != (preTestCmdCount + 1)):
        print("FAIL: Command count did not increment!")
    else:
        print("PASS")


#Using Pytest?
def test_command_counter():
    preTestCmdCount = getCmdCountFromSystem()
    sendCommandToSystem()
    postTestCmdCount = getCmdCountFromSystem()

    assert postTestCmdCount == (preTestCmdCount + 1)

So, correct me if I’m wrong, but it appears that the advantages of using Pytest over plain Python for this simplified case would be:

Being able to make use of pytest’s automated test case discovery, so that I can easily run all of my test functions instead of having to create custom code to do so.
Being able to make use of the ‘assert’ syntax which will automatically generate pass/fail statements for each test instead of having to manually implement pass/fail print statements for each test case

Asked By: tyler124

||

Source

Answer 1

I’ve been in a similar situation, and from what I gathered unit testing frameworks are NOT appropriate for integration testing on embedded systems. A similar question was asked here:
Test framework for testing embedded systems in Python

We personally use Google’s OpenHTF to automate both the integration testing (as in your case) and the production verification testing, which includes bringup, calibration and overall verification of assembly.

Check it out: https://github.com/google/openhtf

We automate advanced test equipment such as RF switches and Spectrum Analysers all in Python in order to test and calibrate our RF units, which operate in the >500 MHz range.

Using OpenHTF, you can create complete tests with measurements very quickly. It provides you with a builtin web GUI and allows you to write your custom export ‘callbacks’.

We’re currently building a complete test solution for hardware testing. I’d be glad to help with OpenHTF if needed, as we’re basing our flagship implementation on it.

Answered By: William Laroche

Answer 2

This thread’s old, but these suggestions might help someone…

For unit testing embedded C on a host PC and on target we use Unity and CMock. http://www.throwtheswitch.org/

For hardware-in-the-loop testing we use pytest.

Both work well and are part of our Jenkins release workflow.

Answered By: Mark Grubb

Should pytest be used for integration testing an embedded system?

Question:

Answers: