Biometric recognition and authentication systems

In this section we focus on error rates and throughput. When considering your own systems, these measures will form part of a wider set of criteria including such things as reliability, availability, maintainability, safety and so on.

In a biometric system, error rates will vary widely depending on the environment in which a system is operating. So accuracy cannot be expressed independently from other factors, such as throughput. For example, if you increase the speed at which a system must authenticate people, you will see a rise in comparison errors. 

For biometrics, there are four types of interrelated errors that must be considered:

• Failure to acquire

Failure to acquire errors are quite common. They come about when the biometric system is unable to find a biometric sample of sufficient quality among the signals gathered. For example, in face recognition systems, a failure to acquire may result from the capture subject not looking towards the camera, from face coverings, or from the system failing to accommodate all the variations in facial characteristics encountered in a population. 
 

• Failure to enrol

Failure to enrol occurs when the system cannot create an enrolment record. This may be due to human factors, such as poor system design or individual physical limitations such as fingerprints  being too faint to register on the sensor. A failure to enrol implies that the individual cannot use the biometric system. Some system administrators avoid recognition errors by choosing not to enrol anyone with poor biometric characteristics. 
 

• False non-match

A false non-match occurs when an individual's biometric characteristic appears not to match their own previously collected biometric characteristic. In an access control system, this results in a false rejection - an enrolled individual is not recognised. An individual that is falsely rejected will be denied access unless an exception handling process is enacted. In a de-duplication system, this can result in the undesired creation of a second record for the individual.
 

• False match

A false match occurs when an individual's biometric characteristic appears to match a characteristic from another individual. In an access control system, a false match results in a false acceptance - an un-enrolled individual is incorrectly recognised and granted access. In a de-duplication system, this can result in the failure to create a record.
 

Inter-dependence of error rates

Where biometrics are concerned, errors are reported as rates, meaning the proportion of attempts to acquire a sample, enrol or access, which result in an undesired outcome. These error rates are dependent on each other as well as on the throughput rates required by the system. In other words, system policies relating to throughput rate, or any of the error rates, will impact all error rates.

For example, as security requirements for an access control system are tightened, authorised persons are more likely to experience a false rejection, but unauthorised persons are less likely to experience a false acceptance.

If the false match rate goes down, the false non-match rate goes up. If the false non-match rate goes down, the false match rate goes up. However, this is only a general trend, increasing one rate does not guarantee significant reduction of the other.

Given the difficulties associated with specifying a required “accuracy” level, it is better to determine the required performance levels for each of these measures:

• Minimum throughput
• Maximum false match rate
• Maximum false non-match rate
• Capacity of the fallback system

You can then compare these requirements to estimates of the performance for a proposed system.

Whether a particular solution will be suitable for your needs often requires iteration over this two step process:

i) Determine an upper bound for the error rates that your system can tolerate given your throughput requirements and fallback capacity. You should seek to avoid over-specifying requirements for the system as this may unnecessarily restrict your choices

ii) Estimate the error rates of the biometric implementation under consideration

In practice, this means you should first try to understand your business requirements. For example, “We would like 90% of individuals to successfully access through the biometric system.” Then you can establish which types of system might meet the performance requirements.

At the design stage it can be difficult to precisely specify required performance, but iterating between stages (i) and (ii) during system development will help you refine your requirements. 

In many ways, this turns the usual security paradigm on its head. Rather than starting with a clear statement of the required security level to which the system must comply, the achievable performance is taken as a starting point for a limited trial. Incremental improvements follow, until either the desired level of security is achieved, or mitigations must be used for any residual gap.

You probably will not have access to test results carried out in a scenario which closely matches your own. This may mean it's necessary to carry out your own tests to determine whether a given product is a good match for your needs. 

Manufacturers tend to test their products under optimal conditions. Unfortunately, most implementations will not reflect these conditions. Consequently, manufacturer claims for biometric system performance are unlikely to match what is seen in day-to-day use. So, what can you do?

Assessing performance results

In some circumstances there may be published performance metrics. For example, the U.S. National Institute of Standards and Technology (NIST) run comparative evaluations for a number of modalities which can be used as a 'first pass' assessment of performance. However, these should be treated with great caution.

As previously mentioned, the performance of a biometric system is specific to its operating environment so it's better to use the results of such evaluations as a starting point, to be followed by further testing of possible solutions in increasingly realistic test environments.

Whatever the source of your data, you should keep in mind the following factors and adjust your expectations according to how these differ from your own situation:

• The choice of biometric data used in the evaluation
• The operating conditions used
• Test subject population
• Desired security posture

Managing errors

To tune a proposed or existing system to your performance requirements you will need to understand how to moderate and trade off the various interrelated factors. 

Fallback

Your exception handling system will deal with rejected persons. It must also be robust enough to separate the occasional unauthorised person, correctly rejected, from the (more common) authorised persons incorrectly rejected. A weak or inefficient exception handling system will jeopardise the entire process.

Throughput

Throughput rate impacts and is impacted by error rates. If the false rejection rate climbs, total throughput rate will decrease. If there is pressure to increase throughput rates during enrolment, both failure to enrol rate, false non-match rate and possibly false match rate will increase due to low quality enrolment biometric measures.

Ergonomics

Throughput and error rates are heavily dependent upon the way data subjects and administrators interact with the system. The importance of ergonomics to the security of the system is regularly overlooked. Ergonomic design and the usability of human-computer interfaces are essential to the successful implementation of any biometric system.

Ongoing performance monitoring

It is important to monitor the performance of existing systems:

• To identity any performance degradation, within and across an enterprise using the biometric system
• The system will require hardware modification and updates during its life cycle and it is crucial that you identify the impact on performance of these changes
• When a technology refresh is required, it is vital to have benchmark data against which you can measure the performance of the new system

There are usually a number of information sources on the performance of a given biometric system. Mostly, these will address only the accuracy of the biometric computer algorithms, or the fidelity of the biometric acquisition devices, which are only two factors contributing to biometric system performance.

When assessing performance results, you should recognise the degree to which the data depends on operating conditions, test subject population, the collection  equipment, and so on.

Some potentially useful sources of performance data are:

• Performance figures from internal vendor testing

Vendor performance figures necessarily make use of a number of assumptions. The most significant is that of an optimal operating environment.

• Existing evaluations by external organisations, e.g. NIST performance reports

The usefulness of such evaluations will be limited by the gap between the test conditions and the proposed biometric system in terms of ergonomics, system components, interfaces with existing databases and business processes.

• Testing from existing implementations

Sometimes there are existing systems which may be sufficiently similar to serve as a useful model – the challenge is to understand what is transferable from one environment to another.

• Internal testing by the prospective purchaser

Your own internal testing is likely to give the best estimate of performance because it's directed at your particular environment and business processes. However, such tests are complex and often expensive.