Software and systems engineering - Software testing - Part 11: Guidelines on the testing of AI-based systems

CSA Group
Standards Development Organisation:
Canadian Standards Association (operating as CSA Group)
Working Program:
View the CSA work program
Designation Number:
ISO/IEC TR 29119-11
Standard Type:
National Standard of Canada - Adoption of International Standard
Standard Development Activity:
New Standard
ICS code(s):
35.080
Status:
Proceeding to development
SDO Comment Period Start Date:
SDO Comment Period End Date:
Posted On:

Scope:

This document provides an introduction to AI-based systems. These systems are typically complex (e.g. deep neural nets), are sometimes based on big data, can be poorly specified and can be non-deterministic, which creates new challenges and opportunities for testing them.

This document explains those characteristics which are specific to AI-based systems and explains the corresponding difficulties of specifying the acceptance criteria for such systems.

This document presents the challenges of testing AI-based systems, the main challenge being the test oracle problem, whereby testers find it difficult to determine expected results for testing and therefore whether tests have passed or failed. It covers testing of these systems across the life cycle and gives guidelines on how AI-based systems in general can be tested using black-box approaches and introduces white-box testing specifically for neural networks. It describes options for the test environments and test scenarios used for testing AI-based systems.

In this document an AI-based system is a system that includes at least one AI component.

Project need:

To align Canadian requirements with those of the respective international standards being proposed for adoption. To maintain alignment between Canadian information and communication technology standards and each respective international standard.

Note: The information provided above was obtained by the Standards Council of Canada (SCC) and is provided as part of a centralized, transparent notification system for new standards development. The system allows SCC-accredited Standards Development Organizations (SDOs), and members of the public, to be informed of new work in Canadian standards development, and allows SCC-accredited SDOs to identify and resolve potential duplication of standards and effort.

Individual SDOs are responsible for the content and accuracy of the information presented here. The text is presented in the language in which it was provided to SCC.