Assessing Biometric Authentication -A Holistic Approach

Biometric authentication is certainly starting to get the attention of the general public. Announcements like the revelation this past fall that over 1 billion stolen passwords had been amassed by a Russian crime ring underscore the fact that the current security systems are flawed, and that new approaches to security are necessary. There is a growing consensus in government and industry (and often confirmed in Hollywood) that biometric approaches are the best path forward. The push by Apple and Samsung to make fingerprint authentication available in their devices is among the most visible applications of biometrics.

Assessment of various biometric offerings however is extremely challenging, even for those with expertise in the field, let alone the average consumer who just wants security without complexity. The biometrics industry knows that there are various aspects that are critical to ultimately determining if a biometric application is a viable product, but it needs to develop better methodologies that allow for an encompassing and holistic approach. Fleshing out the components of such an approach is the first step to having a better comparison framework.

Too Much Focus on False Accepts

The false accept rate (FAR) is a measure of the likelihood that the biometric system will incorrectly accept an access attempt by an unauthorized person. This is generally the most prominent statistic cited for a biometric product, whether in corporate literature or in media coverage of the product.

There is sometimes a nod to the associated false reject rate (FRR) – the likelihood that the system will incorrectly reject an access attempt by the authorized person – as this is directly intertwined with the FAR, and generally a question of fine tuning to adjust the system performance to meet the desired operating range for both parameters. There’s more to cover on the importance of FRR, but first let’s focus on the one that gets too much of the focus – FAR.

FAR is generally determined by collecting the biometric data of a very large set of individuals, and then randomly sampling target individuals and comparing them against the rest of the database. This has led to the creation of very large datasets of fingerprints, irises, faces, and more, some of which are available for public use. Minimizing FAR (traded off against FRR) forms the basis of most research papers in the respective biometric fields, and has spawned various competitions through NIST and other agencies to compare different solutions.

To the extent that the data set is very clean (which is often the case – i.e., good lighting conditions for face, low noise for voice, clean and clear fingerprints), this gives a measure of the inherent uniqueness of the biometric. Fingerprints, for example, have a relatively high inherent uniqueness, which explains in part why they have been so widely used in law enforcement. However, high inherent uniqueness may be offset by other factors in the overall system.

The choice of which data is used to report a system’s performance is, with the exception of the public competitions, highly subjective. It requires an assessment of what constitutes the range and frequency of conditions under which the biometric system will be accessed.

Moreover, in the case of most industry products, it is almost impossible to disprove the claimed accuracies via simple “black box” testing – claims of 1 in 100000 FAR cannot be assessed by asking a few of your coworkers to try break into your phone. As a result, when biometric systems hit the real world, they are often judged directly (e.g., by bloggers) and indirectly (e.g., by non-adoption) by other criteria.

False Rejects are Critical

The false reject rate (FRR) of the biometric system is ultimately critical to user adoption – no matter how secure the system is with respect to possible attacks from unauthorized persons, it will only be adopted if the authorized user is able to gain access a very high percentage of the time.

The system FRR should always be quoted along with the FAR, or else the FAR is meaningless – it is no great feat to design a classifier that always rejects everyone, including the user. It is nonetheless surprisingly common to see only FAR quoted, not only in company literature but also in media articles.

As with FAR, determining the FRR of a given system is highly subjective. It depends greatly on the data chosen as being representative of the range and frequency of conditions under which authentication will be attempted.

There are a number of standard ways to evaluate the combination of FAR and FRR for a given system. Detection Error Tradeoff (DET) curves are a simple plot of FRR versus FAR, which are generated by smoothly increasing the rejection threshold (see Figure 1 for an example DET plot).

At low rejection thresholds, the detection rate (in which the authorized user is allowed access) is relatively high (a low FRR) while the FAR may be fairly high.

Figure 1: Typical DET plot for face recognition

As the rejection threshold is increased (becoming more restrictive), there will be fewer false accepts, at the expense of a lower detection rate (more false rejects). There are other minor variations on this type of plot combining false accept and false reject data, such as ROC (Receiver Operating Characteristic) curves.

One commonly used metric from the DET curve is the Equal Error Rate (EER) – this is the point at which the FAR and FRR are equal.

While EER can sometimes provide a useful quick comparison point, it is important not to rely solely on EER when comparing different biometric systems for at least a couple of reasons. The first reason is that this is often not the point at which the system is intended to operate – rather, the systems are often tuned to run further down the curve at lower FARs. The second and more general reason is that EER of course does not capture the various other critical information that is the basis of the more holistic approach espoused in this article.

FAR and FRR as described above are essentially laboratory measures of the accuracy of a biometric system. Ultimately what really matters to the user is the true likelihood of being able to gain access when using the system in the real world, and the true likelihood that an impostor attack will successfully be thwarted.

Assessing the True Rate of False Accepts

Biometric systems are usually tuned to have very low FARs. As a result, a straightforward FA-based attack, in which random individuals attempt to directly authentic themselves by matching the biometric feature of the authorized user, is highly unlikely to be successful – a lone impostor has a very low probability of having a match.

That lone impostor is also unlikely to have access to thousands of random impostor friends to actually create a higher probability attack. Moreover, most systems put limits in place (such as the number of attempts allowed, or timeouts in between attempts) that make it virtually impossible to actually make thousands of attempts anyway.

Four digit PINs work on this same concept – there are 10000 combinations, so in theory an impostor would be unlikely to find the right combination in any reasonable amount of time. In practice, there are a handful of PINs that are used far too commonly, which makes the likelihood of successful attack considerably higher than 1 in 10000.

Spoofing is Key

The more interesting impostor attack is spoofing, in which the impostor is able to more directly mimic the biometric feature of the authenticated user. This is the likely method that a criminal will use to break in to someone’s device. The specific spoofing method varies from one biometric to another. Fingerprints for example are susceptible to being lifted off device screens and other places, and then recreated with glue, gelatin, Play-Doh or other substances that can mimic the original finger. Face and iris recognition are susceptible to spoofing by images, while voice recognition is vulnerable to spoofing by recordings.

A primary counter attack to spoofing involves “liveness” tests. This can take different forms depending on the biometric being used. In face recognition, for example, motion can be measured to ensure that it is a 3-dimensional face. The challenge-response paradigm is also common – ask the user to do something specific to indicate that a live person is present, like winking or saying a specific phrase.

The downside to a challenge-response system is that it becomes more cumbersome to use, and as a result adoption may be diminished. Many users will be uncomfortable having to wink at their device in public in order to gain access.

Another key counter attack is to require multiple biometrics. This significantly increases the challenge to the attacker by requiring two or more different spoofing methods. The down side is that it may be more burdensome for the user, requiring more than one biometric mode to be presented for every authentication.

Each spoofing method has its advantages and disadvantages – how available the user’s biometric itself is (e.g., fingerprints are left almost everywhere), what level of fidelity of that biometric is required (e.g., how high quality does the voice recording need to be), how much work it will take to create a spoof (e.g., printing an image of the face or iris is relatively easy), and how likely is the counter attack going to be successful (e.g., does the liveness test work?).

All of these contribute to the actual likelihood of a successful impostor attack. The important point is that this should be directly factored into the overall assessment of the biometric system, and is in fact generally more relevant than the basic FAR that is typically quoted.

This often goes unmentioned by the makers of the biometric system, but it does not go unnoticed – the media generally picks up on it very quickly, as has been seen with the fingerprint sensors in Apple and Samsung phones, which were followed almost immediately by announcements of spoof attacks that allowed access.

Assessing the True Rate of False Rejects

The measured false reject rate is highly dependent on the data chosen as being representative of normal usage of the system. Unfortunately this often doesn’t account for the wider variety of conditions seen in the real world. Every biometric has conditions under which authentication can be challenging or even impossible.

For fingerprints, dirt and grease can greatly affect the system accuracy. Lighting can be challenging for face or other camera-based biometrics. Background noise makes voice recognition more difficult. Measuring (and quoting) performance under ideal conditions that don’t represent the real world unfortunately creates unrealistic expectations for how the system will operate. Implementing a system that doesn’t perform well in the real world will only disappoint.

For many biometric systems, the initial enrollment is critical to system performance. An enrollment done poorly or incorrectly can lead to very bad results in a system that might otherwise be capable of very high accuracy. Making sure the enrollment process is as simple and intuitive as possible is key.

Some biometrics lend themselves to adaptive enrollment, in which the user’s enrollment profile can be enhanced over time. This can lead to big gains in accuracy, as the range of environments that are covered can increase, and flaws in the initial enrollment will fade over time.

The amount by which the particular biometric changes over time (sometimes known as permanence) has a strong effect on the true false reject rate. As the user ages, their biometric identity can change. Like environment and enrollment concerns, this too can be mitigated by adaptive enrollment where possible. Universality is also important – that is, does everyone possess this biometric trait? Fingerprints can be lost over time for people in occupations that involve using their hands a great deal. Certain eye diseases can hamper iris recognition.

The true FRR of a biometric system should thus account for the full range and the expected frequency of environmental conditions, the range of enrollment quality that is possible, and the permanence and the universality of the trait.

Factors Affecting User Adoption

There are other factors beyond the inherent accuracy of the biometric system that will affect user adoption, and thus should be considered in any assessment. Acceptability and ease-of-use are two key factors.

Acceptability is a measure of whether the user will be willing to actually use this biometric – if it is embarrassing or invasive, no one will be willing to adopt it.

The acceptability level required may vary from one application to another – for example, the system must be minimally invasive for accessing one’s phone in public, but people are willing to accept a more complicated process for boarding airplanes.

Ease-of-use, along with speed, are really critical to the areas where biometrics are expanding into today. Mobile devices are convenience tools, and users will not adopt something that makes them inconvenient. This is apparent in the low usage rates for basic phone protection with PINs, patterns or passwords – even these are found by many users to be complicated and too slow. Widespread use of biometrics will only occur when using them is fast and easy, including both the training process as well as every day usage.

Cost and Security

Cost is certainly an important issue in consumer devices. Adding sensors specific to a biometric to a device can quickly add substantial cost to retail price. As a result, the fingerprint sensors that are found in mobile devices are only found in high end phones, and yet are of lower quality than those found in dedicated fingerprint systems. Moreover, they are generally smaller and capture less of the fingerprint, which results in lower overall accuracy. Sensor longevity plays into the cost issue as well.

Data security is also critical to creating a viable biometric solution, and has some dependence on the particular biometric involved. One key difference for various solutions is whether or not cloud access is required. Cloud-based biometrics can take advantage of greater computing power and thus potentially push accuracies higher, but at a cost of connectivity requirements, time delays, and most importantly data security concerns. Biometric information for large numbers of users stored in the cloud provide a tempting target for hackers.

In the event of a theft of a biometric, revocability of the biometric is necessary. Just as one can change a password when an account has been compromised, some biometrics also facilitate replacement. One

example is voice recognition, where the passphrase can easily be changed. Unfortunately most biometrics do not easily lend themselves to replacement.

Finding the Right Applications

The concepts that have been described in this holistic approach to assessment of biometric systems – including spoofing, permanence, universality, acceptability and revocability – are certainly not unknown to the biometric research community, or even to the industry in general. But they are often given short shrift in corporate literature and media coverage, and are not easy for the end user to comprehend.

Typically, if these issues are addressed at all, it is in form of a table with simplistic relative rankings like Low, Medium and High, and with little or no supporting information as to how those values were obtained. These factors should in fact be considered explicitly and up front, and this should be done as quantitatively as possible.

Armed with an improved understanding of the real advantages and disadvantages of a particular biometric system, one can then try to assess what are the right applications for that system. The focus is often on high security applications, like banking, but in reality there are applications that span the spectrum of low to high security needs. In all cases, it is important to consider what is currently being used, and whether there is added utility from a biometric system, as opposed to waiting for the perfect biometric system to appear.

The PIN option for locking a phone again provides a good example – it is rarely used, and when it is, it is often one of a handful of likely PINs. Replacing the PIN with a relatively tolerant, easy-to-use biometric can greatly enhance the security in this situation.

As another example, using biometrics as a second factor can also provide much greater security, without being completely reliant on the biometric itself. And in situations where high security is really required, it may also be ok to limit the situations in which the biometric can be used to good environmental conditions, such that the high accuracy can actually be achieved.

It is vital that the biometric industry drives the conversation toward the actual utility that a biometric system brings and helps ensure that the proper expectations are created by presenting a holistic framework in which the real world operation is fairly represented.

About the authors

Gordon Haupt has been building and leading diverse engineering and operations teams for nearly 20 years. He has an extensive background in signal processing and computer vision, and has developed a number of innovative technology products. Gordon is the Senior Director of Vision Technologies at Sensory, where he is focused on bringing speech and face biometrics to consumer devices.

Todd Mozer has worked in the fields of machine learning, speech and vision for over 20 years and has dozens of issued patents in these and related fields. He is the Founder, Chairman and CEO of Sensory.

Sensory is the leader in speech and vision technology for consumer products. Sensory’s award-winning TrulyHandsfreeTM technology offers consumers a voice-controlled, completely hands-free experience, and it can be found in a wide array of popular mobile devices. Sensory has more recently introduced its

TrulySecureTM technology, which combines face recognition and speaker verification. More information is available at http://www.sensory.com.