Information technology - Scalable compression and coding of continuous-tone still images - Part 4: Conformance testing

This document specifies the framework, concepts, methodology for testing, and criteria to be achieved to claim conformance to one or multiple parts of ISO/IEC 18477 as listed below. It provides a framework for specifying abstract test suites and for defining the procedures to be followed during conformance testing.

This document

— specifies conformance testing procedures for decoding of ISO/IEC 18477-1, 18477-2, ISO/IEC 18477-6, ISO/IEC 18477-7, ISO/IEC 18477-8 and ISO/IEC 18477-9
— specifies conformance testing procedures for codestreams to the above International Standards
— specifies codestreams, decoded images, and error metrics to be used within the decoder testing procedures, and
— specifies abstract test suites.

This document does not include the following tests:

— testing decoders for conformance to ISO/IEC 18477-3 only. ISO/IEC 18477-6, ISO/IEC 18477-7, ISO/IEC 18477-8 and ISO/IEC 18477-9 are extensions of ISO/IEC 18477-3 and the required functionality of ISO/IEC 18477-3 is tested as part of the former standards. Testing codestreams for conformance to ISO/IEC 18477-3 is specified in C.2;
— testing codestreams for conformance to ISO/IEC 18477-7 beyond testing them for conformance to individual profiles of this document. Testing such codestreams (full profile codestreams) for syntactical correctness is, however, covered by testing them for conformance to ISO/IEC 18477-3;
— testing of the composition of background and foreground for images reconstructed from ISO/IEC 18477-9 codestreams as this operation is application dependent;
— acceptance testing: the process of determining whether an implementation satisfies acceptance criteria and enables the user to determine whether or not to accept the implementation. This includes the planning and execution of several kinds of tests (e.g. functionality, quality, and speed performance testing) that demonstrate that the implementation satisfies the user requirements;
— performance testing: measures the performance characteristics of an implementation under test (IUT) such as its throughput, responsiveness, etc. under various conditions.
— robustness testing: the process of determining how well an implementation process data which contains errors.

The ISO/IEC 18477 series consists of multiple parts, each of which defines one or multiple profiles. A given IUT (implementation under test) may claim to implement various parts and profiles of ISO/IEC 18477 at once. To test such implementations, they have to be tested with the Abstract Test Suites of each part and profile they claim to conform to.