Coverage

We repeatedly claimed that “coverage” is an important goal of VITRO, but what does this mean? Simply speaking: a measure that tells you what you should test, and how much of it is already tested. For testing the robustness of a computer vision solution for a certain application, two steps are taken by VITRO:

1. Identify scenarios and challenging visual aspects, together with parameters controlling relevant variances
2. Sampling these parameters such that the likelihood of missing an important situation is below a predefined threshold.

For step 1, a systematic application analysis is carried out, together with a hazard analysis, based on the CV-HAZOP catalog. For step 2, different techniques are applied, depending on the nature of the parameter. For those with continuous value range (the very most), sampling with low geometric discrepancy is usually applied, because this optimizes the value domain coverage with minimal sampling points, compared to regular sampling or pure Monte Carlo methods, as indicated in following diagram. It also avoids the risk of missing periodic faults easily caused by regular sampling.

The geometric discrepancy is a measure for the distribution of points in an area A. Formally, it is defined by , with B the subarea, n the total number of points in A and m in B, respectively. See e.g. [Matousek 2010] for a thorough treatment of this topic and sampling methods like Hammersley.