Since its inception in 1999, NIST FRVT (Face Recognition Vendor Test) has been the key global benchmark for the performance of face recognition systems. Through its merits, FRVT has also transformed into one of the most important, inclusive, and insightful benchmarks for AI systems of any kind, tackling challenging topics such as demographic performance while reporting on an enormous array of vendors from around the world. However, we now bid a fond farewell to FRVT while welcoming its replacements, FRTE (Face Recognition Technology Evaluation) and FATE (Face Analysis Technology Evaluation).
In this blog post, we’ll delve into these changes, explore why they are beneficial, and introduce you to the new tests that are set to redefine the landscape of face recognition technology evaluation.
Background
Why has NIST made this change, especially in light of the global reputation NIST FRVT has earned? In short, as technology evolved and expanded, FRVT’s scope had exceeded its designation.
While FRVT started as a way to assess the matching accuracy of leading vendor systems, it evolved into a series of benchmarks that weren’t really assessing face recognition technology. This started with MORPH (which assesses the performance of software designed to detect face morphs) in 2019 and Image Quality (which assesses the performance of face image quality software that estimates the likelihood a given image will produce a false non-match) in 2020. More recently, NIST has added or re-activated tracks for Presentation Attack Detection and Age Estimation.
While each of these is related to face recognition in that they look at images of a face, and may even be critical in the proper design of a face recognition system, they aren’t face recognition technology. They aren’t comparing one face to another for the purpose of determining similarity. And so NIST determined that it no longer makes sense to report on them under the heading of something called a Face Recognition Vendor Test.
And with this in mind, we welcome the new era of NIST FRTE and NIST FATE.
Understanding the change
What is FRTE (Face Recognition Technology Evaluation)?
The NIST FRTE program is all about identifying who is in an image, and covers the traditional FRVT tests, such as 1:1 Verification and 1:N Identification. The move from “Vendor Test” to “Technology Evaluation” is more technically correct as any number of submissions may be from non-commercial parties (i.e., not vendors). The NIST FRTE also includes more specialized face recognition tests, such as Demographic Effects, and Twins Demonstration.
What is FATE (Face Analysis Technology Evaluation)?
NIST FATE on the other hand, takes a different approach by concentrating on what is in an image, and covers all facial analysis categories, such as morph detection, image quality analysis, presentation attack detection, and age estimation. Again, these aren’t face recognition technology, they are analysis of faces images for factors other than gauging similarity.
Conclusion
While we leave the FRVT with good memories, this move by NIST is not only technically correct but can help to set the stage for a better understanding in the media and the public, which often incorrectly conflate technologies such as age estimation or emotion detection with face recognition. In a space which is complex, politically and socially important, and highly nuanced, this type of clarification will help to give policymakers and individuals alike better tools to make informed decisions. Meanwhile, it sets the stage for an evolving technology landscape, where face recognition will remain critical while also complemented by an increasing array of analysis technologies which should be benchmarked independently and designated properly.
More News
