What is face recognition?

Introduction

Over the past fifty years, the unstoppable acceleration of technological development has completely transformed society. The hyper-connectivity enabled by the deployment of communications networks is allowing the digitisation of a significant part of our activities, bringing numerous benefits as well as some risks that need to be addressed. In this analogue-digital duality, one of the processes that remains key to providing security is identity verification.

In the physical world, this is a task we carry out on a daily basis intuitively, recognising people at a glance. When this same (by no means trivial) activity must be carried out automatically without human intervention, it is necessary to deploy all technological potential to obtain reliable results.

In this sense, biometrics in conjunction with artificial intelligence technologies form a robust team that allows us to provide security to digital processes where it is necessary to verify a person’s identity with guarantees.

The aim of this guide is to provide an informative review of the fundamental aspects of facial recognition systems, helping to understand their key points and most interesting applications as well as the most relevant challenges we have to address. 

(Index) 

What is face recognition and what is it for?

In general terms, face recognition can be defined as the process of automatically identifying the identity of a person by analysing an image of his or her face. This process encompasses a whole series of intermediate tasks, from the capture of the digital image to the final decision on the analysed identity.

To cover all workflow stages it is necessary to make use of different technologies, among which biometrics and machine learning are key elements:

• Biometrics: a set of techniques that allows, through the analysis of physical traits (voice, iris, fingerprint, vein pattern, DNA, etc.) or behavioural traits (signature, gait, interaction with digital interfaces, etc.), to determine a person’s identity. Not all biometric traits are equally easy to capture and do not provide the same degree of confidence, so it is necessary to establish a compromise between these factors, taking into account the requirements of the final system.

•  Machine Learning: a branch of artificial intelligence that seeks to make machines capable of making decisions in situations for which they have not been explicitly designed. The problem of biometric identity verification can be tackled purely from the point of view of machine learning algorithms, since it is always necessary to face new examples (new identities) on the basis of a previous training process.

(Index)

Face recognition: A bit of history

It is not easy to define a single milestone for the beginning of the research and development of face recognition solutions, but there is some general consensus to consider Woodrow Wilson Bledsoe’s work in the early 1960s as the starting point.

His approach consisted of recording the spatial coordinates of facial points of reference on a RAND tablet in such a way that a face was characterised by a set of numerical data. When performing an identification process, the biometric features of the input image were compared with those previously stored in the database, so that it was possible to return the one that had the closest similarity.

Although the results were obviously limited by the capabilities of the hardware at the time, these studies determined that facial biometrics was a useful method for identity verification and laid the foundations for the basic workflow in a biometric system: capture, modelling and matching.

Over the next twenty years, progress was slow, limited to an increase in the number of facial features extracted (always anthropometric in nature). It was not until the late 1980s when it was possible to model a facial image in a more robust way.

Algebra methods and the development of machine learning algorithms

Sirovich and Kirby applied linear algebra methods to achieve low-dimensional facial representations, i.e. to reduce the most important features of a face to a small set of numerical values.

This work was extended by Turk and Pentland to apply it to the face detection process, thus opening the door to a fully automatic face recognition process (without the need to provide a previously cropped facial image).

From that moment on, research activity focused on two main lines: on achieving robust facial descriptors and developing machine learning algorithms that would make it possible to find patterns and criteria for distinguishing between these characteristics in order to differentiate between identities.

Facial recognition systems were successfully deployed in situations where it was necessary to automatically verify a person’s identity (such as access control), as well as serving as an alternative to the use of traditional passwords.

Yet, the game changer in terms of the significant increase in the performance of biometric systems has come from the recent boom in deep learning, representing a real paradigm shift and almost unanimous adoption by the industry.

From an ontological point of view, deep learning algorithms are a subset of machine learning, comprising various techniques that seek to obtain high-level abstractions by analysing complex relationships between a large set of input data. A fundamental element of deep learning are neural networks, algorithms with a long history that have achieved full currency thanks to the exponential increase in computational capacity and the availability of vast amounts of data.

It is at this point where we must place the current State of the Art, so the description of the details of the inner workings of a face recognition system that we will carry out use in the following sections will be done from this perspective.

Figure 1: Hardware evolution linked to the development of face recognition algorithms.

(Index)

Download this guide to facial recognition in PDF and discover the origins of the technology, use cases, current techniques and how it will evolve in the future.

The core of a face recognition system

As noted above, from a functional point of view, a face recognition system contains a number of distinct phases that allow the process to be successfully completed:

1. Capture: the beginning of the workflow is to obtain the basic information. In our case this information is just the face portion of the input image, so the first automatic process will be face detection.

In recent years, intensive work has been carried out in the field of face detection algorithms, currently obtaining really high effectiveness rates even in the most demanding environments: artificial lights, very distant positions, use of glasses or masks… The result of this functional block will be a crop of the area of interest from the input image, so that the system can work from that moment onwards exclusively with the relevant information (the facial area).

2. Processing: in any system based on machine learning, processing is a fundamental stage. The objective is to normalise the input data within pre-established parameters so that the performance of the subsequent algorithms is optimal.

Machine learning systems use the conclusions drawn from the analysis of the training set to make decisions about new examples in production environments. For this reason, the input data must move within parameters similar to those used during training, avoiding extreme cases that could lead to undesired results.

In the case of a face recognition system, the processing consists of normalising the input image from the point of view of numerical values (illumination, colour deviations, range of coding values…) as well as content (centring the position of the face around a symmetry axis thanks to the localisation of key facial points).

3. Modelling: this is the key operation in the overall process as it transforms an input image (the output of the processing block) into a set of numerical values commonly known as a feature vector or biometric template. A feature vector could be understood to be a robust encoding of the most significant aspects of a face that differentiates it from other faces.

Artificial intelligence algorithms analyse complex patterns in the input data to locate the features with the highest discriminative power and use them to model an input image.

Currently the feature extraction process is mostly carried out by deep learning algorithms (usually convolutional network architectures), as they are able to find very complex (non-linear) relationships in the input data, far outperforming manual feature extractors used in the previous generation. 

4. Comparison: once the feature vector associated with a facial image is obtained, it is possible to undertake the comparison process. The objective is to determine the resemblance (in numerical terms) between faces in order to implement one of the following biometric operations: Biometric Verification, Biometric Identification or Biometric Matching:

• Verification: is it me? Given an input image and an identity tag, the system must determine whether the person is really who he/she claims to be by analysing the resemblance between the input image and another previously stored in the system. This operation therefore requires a prior registration process.

• Identification: who am I? Given an input image, the system shall determine the identity of the user by comparison with a set of previously registered people.

• Matching: are they the same person? Given two input images, the objective is to determine the match between them. A common use of this operation is in remote customer registration systems, where the image of an identity document is compared with an image taken from the user during the registration process.

  

5. Decision making: the normalised numerical value obtained at the output of the comparison block may not be very descriptive and therefore needs to be contextualised. The objective of the final decision making stage is to return an interpretable response from this numerical value. This response can be a binary value (identity verified/unverified) or a label with the identity associated with the input user in the case of identification.

To make these decisions it is necessary to establish decision thresholds on the numerical values of the comparison in order to achieve a balance between security (very strict thresholds, a very high degree of confidence in the resemblance is needed) and usability (more relaxed thresholds, a lower degree of confidence is needed). Decision thresholds need to be adapted to the needs of each project by assessing the cost of a higher incidence of misclassification errors:

• False positive: incorrect verification of a person’s identity. This is a mistake that has a high cost in security applications, where letting an impostor through is a serious threat.

• False negative: incorrect rejection of a person’s identity. This is an error that has a high cost in forensic applications, where wrongly rejecting a person’s identity can be a serious problem.

(Index)

Use of face recognition technology

A biometric system based on face recognition has multiple applications where it is required to verify the identity of users in digital environments. These are some of the most interesting and widely adopted by the industry:

• Digital onboarding: remote customer registration processes require reliable verification of a person’s identity. For this purpose, facial recognition provides a useful tool not only to verify identity but also to ensure that there is no fraud, such as synthetic identity theft. Two examples of the integration of a digital onboarding system with facial recognition in the financial sector are Banco Continental’s Digital Account and Akisi App’s e-wallet.

• Access to digital operations: during the digital onboarding process it is possible to securely store the user’s biometric traits so that this information can later be used to control access to different operations securely: digital payments, access to transport, collection of prizes in online gaming rooms, etc.

• ATMs: In recent months, many ATMs have been incorporating digital cameras that allow the biometric identity verification process to be performed in a simple way. This brings agility to transactions, as well as adding an extra layer of security by preventing the possible criminal use of bank cards (either by replacing the traditional PIN code or by providing two-factor authentication).

• Time and attendance control: new regulations regarding time and attendance control open the door to massive use of biometrics as an agile method of registering or clocking in during the working day, while preventing fraudulent practices. C

• Online check-in: it is possible to digitalise the customer registration process in hotel establishments, obtaining all the necessary information that can also be used to manage access to rooms by means of biometric verification subsequently.

• Education: by using biometric verification, universities and academies can control biometric access to classes and rooms. It ensures that only authorised people participate in sessions. Furthermore, biometric authentication enables accurate attendance, which facilitates the evaluation of progress and compliance with training requirements.

• Event management: facial recognition can facilitate access to events with large crowds, such as concerts, festivals or conferences. Attendees can register and then enter the event using biometrics, simplifying the entry process and improving security by preventing unauthorised access.
• Passenger flow: implementing this technology eliminates the need to present physical documents and streamlines the flow at counter check-in.

• Identity validation in delivery and logistics: facial recognition can be used to verify the identity of parcel recipients. It ensures that goods are delivered securely and to the right person.

• Facial recognition in social media: biometric technology can facilitate users’ access to their profiles and, at the same time, boost account protection and help age verification to prevent access by children.

(Index) 

Facial recognition and society

The possibility of carrying out identity verification using biometric methods offers great advantages, but like any other technology based on human-machine interaction, it involves some challenges when we analyse its social implications.

Given that this type of tool can be used in many different contexts (some of them critical), it seems necessary to reflect on the most relevant aspects involved in its use, including the advantages and challenges related mainly to the field of security.)

(Index) 

Authentication beyond passwords

Historically, passwords have been the most widely used authentication method to verify people’s identity in digital environments. We are all forced to remember different sequences of characters that prove that we are authorised to carry out different operations: bank account management, access to email, access to private environments, etc. This requires us to use passwords that, in order to be secure, must be complex and frequently updated.

Indeed, this is its weakest point. For convenience reasons, it is common to use the same password for different services or to use a simple and easy to remember one, increasing the possibility of being hacked and used for fraudulent purposes.

Through its application in identity verification systems, biometrics simplifies and strengthens the process, moving from something we know (the password) to something we are (our biometric features). The integration of facial recognition systems in business solutions brings interesting benefits in three main areas:

• Agility: it is possible to automate the entire process (from capture to decision making), saving time in operations and improving the user experience by avoiding complex passwords.

• Security: biometric technologies have proven to be a robust method for determining a person’s identity with guarantees. Furthermore, we remove the threat of password theft.

• Modularity: the technology can be easily integrated into existing applications and can be used in conjunction with classic password-based methods to implement two-factor authentication systems.

(Index)

Face recognition security challenges

Secure and reliable systems

Security (understood from a general point of view) is a key element in biometric recognition systems. Both the reliability of the results and the trust involved in the process depend on it. In order to successfully implement a biometric system in a commercial solution, security threats must be minimised.

Therefore, the analysis carried out in ISO/IEC 30107-1 regarding the main potential attack points in a generic biometric system is very useful for this purpose.

Although the threats may seem too many, it is possible to group them into three blocks according to their nature:

• Presentation attack: this is an attack to the biometric capture device with the aim of interfering with the correct functioning of the system. Its detection is crucial in a biometric solution and will be discussed in more detail in the following section.

• Attacks on processes: each of the functional blocks is susceptible to attack with the aim of modifying its operation. Examples of such actions could be the replacement of the model used for feature extraction (rare) or the modification of decision thresholds to interfere with the system and make it more permissive. 

• Attacks on communication channels: any connection point between blocks can be intercepted either to modify the information being transmitted or to inject different information. A clear example could be the substitution of the feature vector of the comparison image to obtain different results or the modification of the numerical value of the comparison to interfere in the final decision.

Attacks on processes or communication channels are not very common as they require a high degree of sophistication as well as a deep understanding of the details of the given biometric system. Even so, it is necessary to establish strict security policies that prevent unauthorised access to the app, while trying to minimise the exposure of key resources in the operation of the system.

Check out the following article for more information on how we apply artificial intelligence to detect identity document fraud

At this point it is interesting to mention data protection, the raw material of the biometric system. The information used in a facial recognition process is considered as a special category by the General Data Protection Regulation (GDPR) and therefore subject to the highest protection given its personal nature.

It is therefore necessary to work with the utmost care in information security policies, ensuring that even in the case of a data leak, it will not be possible to recognise individuals in a biometric database by reconstructing the original features (facial image) through reverse engineering.

In this regard, facial modelling algorithms based on deep learning provide great robustness, as it is difficult to recompose the original image from the one generated by the feature vector (hence they are also known as unidirectional).

The combination of this type of algorithm together with the encryption of all the information generated, enables the implementation of secure systems that meet today’s strict regulations.

(Index)

The threat of identity theft

The most common type of attack that a biometric system can face is the presentation attack: a user acting on the capture module with the intention of spoofing another person or concealing his or her own identity. In most cases, the purpose of the attacker is to supplant another person’s identity without authorization, which is why it is common to replace the general term “presentation attack” with the more specific term “spoofing attack”.

The capture subsystem is the most vulnerable element as it is the most exposed. For this reason, it is necessary to focus all efforts on detecting all suspicious actions. An important concept with regard to spoofing attacks is the attack instrument (i.e. the biometric feature or object used to carry out the attack).

In the field of facial biometrics, it is common to use attack tools such as masks, images or recordings showing the face of the person to be impersonated. The objective of an automatic spoofing attack detection system will therefore be to analyse the input information to detect traces of the use of such tools.

This process is by no means trivial and represents a huge challenge, especially when it comes to generalising the operation of attack detection algorithms to the real environment, where we will encounter a wide variety of situations that are sometimes difficult to control.

Deepfakes detection

Deepfakes are audiovisual manipulations that use deep learning techniques to create fake content, often to impersonate a person in a video. This manipulation presents a significant challenge in detecting impersonation attacks, as they can potentially fool facial recognition systems.

It is essential to analyse multiple aspects of the multimedia content to identify deepfakes. It includes assessing visual and temporal inconsistencies, such as anomalies in facial appearance or unnatural movements. Checking for consistency with the context and environment in which the video is supposed to have been recorded is also crucial. Furthermore, comparing biometric features can provide valuable clues about the authenticity of the content.

Some of the measures implemented by spoofing attack detection systems can be grouped into two blocks according to the need for interaction with the user:

In the final process it is necessary to combine the information from both measures to return a result about the probability of attack. Then this is combined with the result of the matching between the document image and the one extracted from the capture device to make a final decision about the whole identity verification process, providing confidence about the process.

(Index)

Face recognition: Ethics, bias and accessibility

When analysing the social implications of the use of face recognition technologies, the ethical factor is crucial. As with any disruptive technology, there are great opportunities for process improvement, but there are also risks that self-interested use could have negative effects.

There is latent concern about the use of face recognition systems that could lead to excessive social control by some governments (as in the case of China and its citizen’s points-based ID card), as well as the not always transparent use of facial biometric algorithms by private companies with the aim of obtaining the maximum information from their users.

In this sense, there is a need for clear regulation regarding the permitted uses and limitations in terms of personal data protection policies, which are already being implemented in many Western countries.

Biometric operations in the privacy and data protection framework 

When we talk about the processing and treatment of biometric data, it is vital to consider the technique used, the specific nature of the processing, its scope and, most importantly, the objectives pursued because biometric operations can have different impacts on privacy and affect individuals differently.

Biometric data processing techniques are based on capturing and analysing physical features, making identifying individuals possible.

Each biometric technique must be assessed attending the principle of necessity, appropriateness and proportionality, its purpose, and its impact on individual rights.

It is important to note that according to the General Data Protection Regulation (GDPR), the user’s explicit consent is required, and an impact assessment of the system must be performed.

Demographic bias and facial recognition

Another relevant aspect, with a major social dimension, is the need to create accessible and fair biometric systems. Machine learning algorithms are trained with thousands of examples stored in databases, so from the very beginning their learning is conditioned by the nature of this data.

Although face recognition systems have been extensively tested, the performance is not uniform for all users, in that there is a demographic bias in the results.

There are different variables (gender, ethnic group, age, etc.) which, depending on their greater or lesser representation in test databases, lead to more or less accurate behaviour in the results. To avoid (or try to mitigate as much as possible) this type of bias, vendors must ensure maximum representation from a demographic point of view in their databases, providing universal access to their systems.

(Index)

A look into the future of face recognition technology

In a constantly evolving industry such as biometrics, it is difficult to make long-term forecasts, but it is true that there are certain consolidated trends in the sector that allow us to foresee what major trends will emerge in the near future. 

The artificial intelligence dialectic

Artificial Intelligence is undoubtedly one of the main enabling technologies that within the industry 4.0 paradigm allow further development of the processes and services offered by technology companies. However, this growing presence also implies a growing risk of non-legitimate use, as these technologies can be used to create sophisticated attacks that are increasingly difficult to detect.

In the field of facial recognition, there is evidence of the use of artificial intelligence techniques to generate artificial biometric features (commonly known as deep fakes) or even to try to regenerate the original trait from the feature vector. This struggle between stakeholders forces vendors to stay one step ahead of attackers by implementing working methods based on continuous improvement to maintain the reliability of systems.

(Index)

Face recognition regulation

One of the most appreciated trends in recent years in the business sector is the digitisation of processes involving interaction with customers, encouraging a shift towards unattended models.

At the same time, regulations have gradually emerged to regulate the digital activities of companies in key sectors such as banking, with the aim of preventing potential abuses such as money laundering or tax evasion. 

An example of this is the European PSD2 Directive, which aims to regulate payment services within the EU in order to provide security and transparency, as well as to promote competition and innovation in payment services in the financial sector.

On a similar line, at the beginning of 2020, the AML5 Anti-Money Laundering Directive came into force, which seeks to impose on companies, in different sectors, a series of control measures to prevent fraudulent conduct, by always identifying the interveners. One of the key steps in implementing AML solutions is the KYC (Know Your Customer) process.

The purpose of KYC is to collect all of a client’s information at the time of contracting a service, so that the identity of the user is reliably established. A key element in that process is therefore identity verification, so facial recognition can be an integral part of this process.

(Index)

Mobbeel face recognition solutions

Since our foundation in 2009, Mobbeel has developed products for identity verification. We were pioneers in applying biometric recognition technologies to mobile environments using only the standard sensors of a mobile device. 

A large number of top-tier clients in different sectors, such as banking and finance, insurance, health, telecommunications and online gaming, rely on our technology to validate their clients’ identity in a variety of scenarios.

From the first moment that they begin their relationship through identity verification in a digital onboarding process, the multi-biometric recognition each time they access the system or authorise a transaction, to the biometric signature to validate legal contracts or agreements.

Face recognition functionality is included as a core element in two of our products:

• MobbScan: our digital onboarding solution that complies with KYC and AML regulations. It allows the extraction and validation of personal information obtained through the analysis of identity documents as well as secure identity verification through facial recognition. MobbScan is flexible and modular, and can be adapted to the needs of each specific project. It currently supports more than 250 different documents (ID cards, driving licences, passports) from 194 countries, with the possibility of easily adding new document types.

• MobbID: a multi-biometric identity verification solution that allows validation of the identity of customers through the analysis of one or several biometric features such as face, iris, fingerprint, voice or biometric signature, thus integrating identity fraud controls. The process can be carried out both online and offline, allowing it to be used even in low connectivity conditions. Like MobbScan, MobbID also allows a flexible and modular implementation based on SDKs.

Download this guide to facial recognition in PDF and discover the origins of the technology, use cases, current techniques and how it will evolve in the future.

NIST’s role in facial recognition assessment

The National Institute of Standards and Technology (NIST) is the US Department of Commerce agency charged with promoting innovation and industrial competitiveness by setting standards and quality measures.

In facial recognition technology, NIST plays a crucial role in providing impartial and rigorous assessments of facial recognition systems, which contributes to improving and standardising this technology.

Its testing and analysis help the industry to develop more accurate and secure systems while providing users and regulators with valuable information on the performance of these solutions.

Performance assessment of Mobbeel’s biometric algorithms

The assessments of this body involve vast testing of facial recognition systems provided by a wide range of technology providers, including Mobbeel.

Facial recognition algorithms are evaluated for different purposes, including analysing the quality of these algorithms and ranking them according to their accuracy and efficiency in different scenarios.

These assessments involve rigorous testing in different scenarios, ranging from ideal conditions to challenges with low lighting and variations in facial expression, among others.

NIST has assessed Mobbeel’s 1:1 facial verification technology (MobbID).

To illustrate this, the ICT Security Guide CCN-STIC 140, which is used in the certification of video identification tools, specifies that the biometric module must achieve in the VISABORDER configuration a false negative rate (FNMR) of less than 5% at the set working point which equals a false positive rate (FMR) of 0.0001%.

In this category, Mobbeel has achieved an FNMR value of 0.99%, which means that our technology is five times more effective than the minimum required by the National Cryptologic Centre for this type of application.

It is necessary to mention that the assessments performed by NIST are not limited only to face matching 1:1. They also extend to face identification 1:N, which is the process of identification where seeking an identity among a set of people, such as in face databases.

Our technology has received a positive evaluation by NIST in the 1:N category. This distinction means that our solution has been validated to perform face identification operations on sets of previously registered identities.

This capability is essential in versatile projects that demand the identification of individuals within a larger database, giving our users the confidence and security needed to perform face identification operations effectively and accurately.

  Download the face recognition guide as a PDF document

What will you discover in the face recognition guide?

• Learn about the history of facial recognition. Origins and evolution.
• Learn how facial recognition systems work.
• See the different uses and applications.
• Understand the different implications of the technology in our society today.
• Discover the challenges in cybersecurity.
• Learn about the regulations involved in biometric processes.