Validation of Test Specifications

A standardized conformance or interoperability test specification should be validated using similar methods to those for other standards. Such a specification typically comprises:

• a set of Test Purposes;
• the definition of a Test Suite Structure and of the implementation conformance/interoperability statement and the extra information for testing (often defined textually);
• an Abstract Test Suite (ATS) which is most often specified in TTCN-3. 

Validation of Test Purposes

The formalized design review method of validation can be quite effective in validating the Test System Structure and the Test Purposes as they generally use a mixture of textual and graphical specification techniques. Such a review will easily highlight gaps in the test coverage of the base requirements as well as any unnecessary testing of unspecified functions. It will also validate that the general approach to testing, the test configurations and the individual test methods implied by the Test Purposes are all valid.

Test Suites, however, cannot easily be validated by means of a design review as they are written in TTCN-3 or some other programming language. It is, of course, possible to partially validate the structure and coverage of a test suite by such a manual method but the correctness and effectiveness of a TTCN-3 test suite can only be validated by execution of the programming code in either a simulation or a controlled testbed.

Validation of Test Suites

In order to produce high quality executable test suites some form of practical validation must be done before they are published as ETSI standards. The following definitions of validation levels for standardized TTCN-3 test specifications have been developed by the ETSI Centre for Testing and Interoperability (CTI). The intention here is to provide guidance to Technical Bodies on:

• the selection of a validation level for a particular test suite to be developed;
• the completion of the validation aspects within the Terms of Reference (ToR) for a Specialist Task Force (STF) to develop the test suite;
• the construction of a validation report

There are three levels of validation defined for TTCN-3 test specifications, as follows:

Level 1: The test suite successfully analyses/compiles on one or more TTCN-3 test tools. Optionally, a design review of the test suite is performed.  Level 2: The tests are executed on one test platform against at least one System Under Test (SUT). Where possible, tests will be run to completion. Optionally, a back-to-back validation (mirror test cases) and/or a data-driven codec validation are performed. Level 3: The tests are executed on several test platforms against several SUTs. Tests will be run to completion and trace analysis will be done.

Further details on the validation processes for the three levels are given in Annex B of EG 201 015.

Each validation level carries its own overlapping set of responsibilities that need to be considered and these are identified in the table below.

 

Rigorous Validation (Level 3)
Strong Validation (Level 2)
Basic Validation (Level 1)
Provision of TTCN-3 analyser/compilers
Compilation of the test suite
Reporting of errors in the tools
Reporting of errors in the ATS
Updating of ATS
Compilation of validation report
	Setting-up and maintaining the test platform
	Provision and installation of SUT(s)
	Execution of the tests
	Reporting of errors in the platform
	Validation of test verdicts
		Analysis of output traces