STFbreadcrumb separatorSTFsbreadcrumb separatorSTF Homepagesbreadcrumb separatorSTF370

Specialist Task Force 370:
Automated Interoperability Testing for Distributed Systems

Who we are:

Team Leader: Stephan Schulz
Team Members: Anthony Baire
Alexandre Berge
Olaf Bergengruen
Francois Fischer
Yann Garcia
Tuncay Namli
Bostjan Pintar
Thomas Rings
Luca Seracio
Tomas Urban
Theofanis Vassilou-Gioles
Peter Kremer
Sebastian Mueller  

  

Why we do it:

This activity aims to capitalise on past ETSI experience in conformance and interoperability testing and validation expertise by developing a full and coherent generic methodology, framework validated by a prototype IMS case study platform that combines interoperability testing with conformance checking. Other technologies such as GRID, IPv6, WiMax, and B2B will into be taken in account for methodology development.

How we do it:

A trial based on IMS will be carried out to validate the methodology developed in this project. Methodologies are only successful if they are well understood and applied and therefore this project will also develop training materials supporting the promotion of these approaches. IMS has been chosen as the case study because of the maturity and stability of the standardisation taking place in this area. The lack of coherent pan-European standardisation of GRID and B2B currently makes it impractical to base the case study on these technologies. However, ETSI TC GRID (for example) is extremely keen to liaise with the project and to contribute to the review and development of the methodology.

Testing the conformance of every single entity (protocol layer) in a complex system such as those deployed in an IMS network is almost prohibitively expensive. On the other hand, simply performing top-level interoperability tests does not give the required degree of thoroughness. Interoperable systems may still be non-conformant, and vice versa. For example, in a recent study ETSI demonstrated 90% interoperability between systems but with only 60% conformance. In the field, this would undoubtedly lead to interoperability problems.

However, over the past few years a combination of the two approaches is giving promising results: we call this interoperability testing with conformance verification. The basic idea is simple: end-to-end interoperability tests are executed while at the same time key reference points (interfaces) are checked to see that the message flow conforms to the flows mandated by standards.

The eventual verdict (pass or fail) is set depending on the outcome of both the interoperability and the conformance criteria.

This is illustrated in the figure below.

The focus of this project is the development of a generic methodology for the automated interoperability testing with conformance checking of complex distributed systems such as IMS. The proof of concept of the methodology will be done through a case study in the domain of IMS. These case studies are intended to provide a proof of concept in the two different application domains. There is no direct relationship between the two case studies which are intended to be separate case studies for applying the same methodology.

For more details, see our Terms of Reference

Time plan for the work:

The STF will be reporting the milestones to the ETSI Technical Bodies Methods for Testing and Specification (MTS), IP Multimedia Subsystem Network testing (INT), and Grid. The work will proceed according to the following time schedule:

  • The project has started work on February 23rd 2009
  • First draft of methodology and progress report is planned to be made available for review in week 20
  • Presentation of first working results at TTCN-3 User Conference at Sophia, FR, June 4/5
  • First draft of IMS architecture and progress report is planned to be made available for review in week 24
  • First IMS TTCN-3 IOT ATS, codec, and adapter implementation and progress report is planned for week 32
  • First application of methodology and TTCN-3 test system at IMS Plugtest 3 in Lannion, FR, week 44
  • Plugtest and Progress Report is planned to be made available for review in week 45
  • Presentation of project status at FOKUS IMS Workshop in Berlin, DE, week 46
  • EC/EFTA Interim Report planned for week 49
  • First drafts for white paper and training material are planned to be made available for review in week Q1/2010

Final drafts are to be submitted to ETSI MTS/INT for approval by June 2010

How to contact us:

You are welcome to send an email to : stephan.schulz@etsi.org

 

This information is based upon STF working assumptions.
The views expressed do not necessarily represent the position of ETSI in this context.

Last updated: 2013-04-13 17:31:03