Test execution engine

A test execution engine is a type of software used to test software, hardware or complete systems.

Synonyms of test execution engine:

• Test executive
• Test manager
• Test sequencer

A test execution engine may appear in two forms:

• Module of a test software suite (test bench) or an integrated development environment
• Stand-alone application software

Concept

The test execution engine does not carry any information about the tested product. Only the test specification and the test data carries information about the tested product.

The test specification is software. Test specification is sometimes referred to as test sequence, which consists of test steps.

The test specification should be stored in the test repository in a text format (such as source code). Test data is sometimes generated by some test data generator tool. Test data can be stored in binary or text files. Test data should also be stored in the test repository together with the test specification.

Test specification is selected, loaded and executed by the test execution engine similarly, as application software is selected, loaded and executed by operation systems. The test execution engine should not operate on the tested object directly, but through plug-in modules similarly as an application software accesses devices through drivers which are installed on the operation system.

The difference between the concept of test execution engine and operation system is that the test execution engine monitors, presents and stores the status, results, time stamp, length and other information for every Test Step of a Test Sequence, but typically an operation system does not perform such profiling of a software execution.

Reasons for using a test execution engine:

• Test results are stored and can be viewed in a uniform way, independent of the type of the test
• Easier to keep track of the changes
• Easier to reuse components developed for testing

Functions

Main functions of a test execution engine:

• Select a test type to execute. Selection can be automatic or manual.
• Load the specification of the selected test type by opening a file from the local file system or downloading it from a Server, depending on where the test repository is stored.
• Execute the test through the use of testing tools (SW test) or instruments (HW test), while showing the progress and accepting control from the operator (for example to Abort)
• Present the outcome (such as Passed, Failed or Aborted) of test Steps and the complete Sequence to the operator
• Store the Test Results in report files

An advanced test execution engine may have additional functions, such as:

• Store the test results in a Database
• Load test result back from the Database
• Present the test results as raw data.
• Present the test results in a processed format. (Statistics)
• Authenticate the operators.

Advanced functions of the test execution engine maybe less important for software testing, but these advanced features could be essential when executing hardware/system tests.

Operations types

A test execution engine by executing a test specification, it may perform different types of operations on the product, such as:

• Verification
• Calibration
• Programming
  • Downloading firmware to the product's nonvolatile memory (Flash)
  • Personalization: programming with unique parameters, like a serial number or a MAC address

If the subject is a software, verification is the only possible operation.