Conformance Testing

The purpose of conformance testing is to determine to what extent a single implementation of a particular standard conforms to the individual requirements of that standard.

The important factors which characterize conformance testing are as follows:

• the System or Implementation Under Test (SUT or IUT) defines the boundaries (open interfaces) for testing;
• the conformance test system is a specialized tool (system) built for the purpose of testing and on which specific test scripts can be run; 
• the SUT comes from a single supplier (or, at least, a single product line); 
• the tests are executed by a dedicated test system that has full control of the SUT and the ability to observe all communications from the SUT; 
• the tests are performed at open standardized interfaces that are not (usually) accessible to a normal user (i.e. they are specified at the protocol level); 
• the tests are specified at the detailed protocol level and are not usually based on functionality as experienced by a user.

Because the conformance tester maintains a high degree of control over the sequence and contents of the protocol messages sent to the IUT it is able to explore a wide range of both expected and unexpected (invalid) behaviour.

Development of ETSI Conformance Test Specifications

Most ETSI test specifications are developed according to the well-proven methodology defined in ISO/IEC 9646. This standard, which covers the entire test development process, provides an excellent basis for the production of high-quality test frameworks and specifications.

The following diagram shows the development of test specifications process, acronyms are explained below.

Building a coherent set of test specifications from disperse requirements sources can be simplified by gathering the requirements together into a single catalogue. A Requirements Catalogue should list all implementation requirements from the various sources and organize them into an appropriate structure.

The Implementation Conformance Statement (ICS) is a checklist of the capabilities defined in the standard. When filled in by the user, the ICS provides an overview of the features and options that are implemented by any given IUT. The ICS can be used to select and parameterize test cases and as an indicator for basic interoperability between different products.

The Implementation eXtra Information for Testing (IXIT) contains additional information (specific addresses, timer values etc.) necessary for testing.

In the specific case of protocol standards the ICS is known as the Protocol ICS or PICS and the IXIT is known as the Protocol IXIT or PIXIT.

Test Purposes (TP) provide an informal, easy-to-read self-contained description of each test, concentrating on what is to be tested rather than how the actual test may be achieved. They are derived from the relevant base standards or from the appropriate Requirements Catalogue. The set of all Test Purposes is grouped into a logical Test Suite Structure (TSS) according to suitable criteria (basic interconnection, error handling, functionality etc.). The standardized notation TPLAN is a good way to write consistent Test Purposes.

The Abstract Test Suite (ATS) is the entire collection of Test Cases, sometimes known as test scripts. Each Test Case specifies the detailed coding of the Test Purposes, written in the standardized test specification language TTCN-3. The actual Executable Test Suite (ETS) can be quickly and easily implemented from the ATS using the TTCN compilers available on many test tool platforms.

Finding ETSI Test Specifications

ETSI test specifications can be downloaded via the ETSI Work Programme application.

Keywords can be added to facilitate your search, the following are related to testing:

• ATS (Abstract Test Suite) 
• TTCN PICS (Protocol Implementation Conformance Statement) 
• ICS (Implementation Conformance Statement) 
• TSS&TP (Test suite structure and Test purposes) 
• Testing 
• Conformance