Performance Evaluation of Biometric Voice Verification Tools with Identical Twins

This strong signal not only contains the basic frequency of vibration, which we normally perceive as the pitch of speech but also its harmonics. The dimensions of the vocal tract and larynx vary from person to person, and it is these variations that combine to give each individual his or her own voice. In addition, the manner in which speakers control their vocal tracts contributes characteristics that are acknowledged as accent and personality. All this means that although the basic mechanism appears simple, there is only a remote possibility of a particular signal ever being identically reproduced, even by a single speaker repeating a word.

The objective of this work is to evaluate the voice biometric speaker verification products with identical twins, subject their voice samples for enrollment, authentication and as an imposter of their counterpart. We were not able to acquire the Software Development Kits (SDKs) of the prominent voice biometric products to simulate the test with exact phrases for enrollment, we, therefore, chose the closest matching phrases available on their demo platforms. The Exception was VoiceVantage, as its Parent Company, THC Technologies Corp provided voice collection and voice verification tools which facilitate the capture of phrases of our choice and analyzes the similarity scores in detail.

The performance of a voice verification system that is solely based on a global analysis can be greatly improved if a local analysis is incorporated into the decision process. While Coarse Measurement performs very well, there are several challenges that currently face all available voice verification systems. The first challenge is that a frame is not related to a speech event. In fact, from a speech perspective, a frame is rather arbitrary. Meaning that sometimes a frame captures a relevant piece of voice speech (i.e., pitch period), and sometimes it captures irrelevant information such as silence, or outside noise. That means that all systems capture information that actually makes voice verification more difficult.

A number of approaches have been tried to counteract this universal effect. Some verification systems try to weight some frames more heavily than others, while other systems attempt to discard frames that have irrelevant information. Both of these approaches tend to throw away relevant information and keep irrelevant information. The ideal approach is to find all the pitch periods in a phrase and vary the size of the frame to fit each pitch period exactly. By zeroing in on only the most important information, the system becomes significantly more accurate.

The next challenge is that most verification systems do an analysis essentially based on an outline of the rich frequency or pitch information in the voice. While in most cases, this outline is enough, there are many times where more information is needed. Currently, most systems try to get a second outline, other systems simply fail.

Fine Measurement takes voice verification analysis to a new, deeper level. Imagine looking at a full-color photo rather than simply an outline. Essentially, the new technology breaks down the frequency information into all its component parts, allowing the system to analyze all of the rich details of the human voice.

Fine Measurement solves both of the most substantial technical problems facing voice verification systems today. It not only zeroes in on the most important information in the human voice, but it also analyzes that information at a deeper level than ever before.

Enrollment and identity verification are the two fundamental processes involved in voice verification. The enrollment process involves creating the database of voiceprints that can be referred to in order to compare and verify with samples for identification later. The enrollment process takes less than a minute for an average user to finish. Verification is the process of comparing a live voice sample to the previously enrolled voiceprint. As mentioned earlier, verification includes end-pointing, analysis, comparing and scoring.

Dynamic Time Warping (DTW) algorithm was developed specifically for speech recognition which aims to align multiple sequences of feature vectors by warping time axis repetitively until an ideal match as per the suitable metrics between those sequences is identified.

Consider two sequences of feature vectors:

The two sequences can be arranged on the sides of a grid, with one on the top and the other up the left-hand side. Both sequences start on the bottom left of the grid.

Inside each cell, a distance measure can be placed, comparing the corresponding elements of the two sequences. To find the ideal match between these two sequences, we need to determine a path through the grid which minimizes the total distance between them. The procedure for calculating this overall distance involves identifying all possible routes through the grid and for each one to compute the overall distance.

The overall distance is the minimum of the sum of the distances between the individual elements on the path divided by the sum of the weighting function. The weighting function is used to normalize the path length. It is obvious that for any considerably long sequences the number of possible paths through the grid will be extremely large. The major optimizations or constraints of the DTW algorithm arise from the observations on the nature of acceptable paths through the grid. The constraints include:

The above constraints allow restricting the moves that can be performed from any point in the path and so limit the number of paths that need to be analyzed. The key advantage of this algorithm is that in spite of identifying all possible routes through the grid which satisfy the path constrains, the DTW algorithm computations are based on keeping track of the cost of the best path to each point in the grid. While computing the grid, the minimum overall path is unknown, but it can be backtracked once the endpoint is reached.

Linear Predictive Coefficient (LPC) is one of the most powerful speech analysis techniques and efficient for encoding quality speech at a low bit rate. The basic idea behind linear predictive analysis is that a specific speech sample at the current time can be approximated as a linear combination of past speech samples.

LPC is a model based on human speech production. It utilizes a conventional source-filter model, in which the glottal, vocal tract, and lip radiation transfer functions are integrated into one all-pole filter that simulates the acoustics of the vocal tract.

The LPC analysis of each frame also involves the decision-making process of voiced or unvoiced. A pitch-detecting algorithm is employed to determine to correct pitch period/frequency. It is key to re-emphasize that the pitch, gain and coefficient parameters will be varying with time from one frame to another. In reality, the actual predictor coefficients are never used in recognition, since they typically show high variance on each occurrence. The predictor coefficient is transformed into a more robust set of parameters known as cepstral coefficients.

Bit rates, overall delay of the system, computational complexity and objective performance evaluation are the parameters that are involved in the performance evaluation of LPC.

Voice-excitation, residual excitation, pitch excitation, multiple excitations, regular pulse excitation, and coded excitation are important types of LPC.

A wide range of voice biometrics products are available on the market today and we have decided to choose the below products including VoiceVantage, Verint-VoiceVault, Microsoft- Azure Speaker Recognition, and now Microsoft owned Nuance, to evaluate their performance with identical twins. As THC has provided their VoiceVantage VoiceCheck™ 32-bit and 64-bit tools for evaluation, we were able to conduct in-depth twin tests for VoiceVantage, whereas the twin tests for other products were brief with the viable limited options provisioned on their demo platforms.

Comparing the sample with the voiceprint consists of several steps like end pointing, analysis, comparing and scoring. End pointing is about identifying the exact beginning and end of the phrase on the voice sample. Each phrase will be recorded multiple times during enrollment and each of them will be compared with the voice sample to get the score. The analysis is based on Linear Predictive Coefficient (LPC) Cepstral Analysis technique by slicing the voice sample and the print into 30 ms frames and comparing those slices to get the score.

Scoring is the process of calculating the difference between the file being presented and the file in the database to create a similarity or confidence score. Both the comparing and scoring are achieved using the Dynamic Time Warping (DTW) method. The “live” speech is compared to each of the stored versions of that phrase and the best score is identified.

VoiceCheck™ uses a text-dependent voice verification system in which a customer’s voice is enrolled in the system by creating and recording algorithmic examples of the customer’s speech (usually words or a text phrase) for use as a reference. When verifying identity, the customer is prompted to speak the same words or phrases which is then compared to an earlier stored version. The two signals are compared based on signal timing, inflection, and variance. If the new word or phrase is sufficiently similar to the previously stored version, the claimed identity is accepted. If not, the person is assumed to be an imposter and the identity claim is rejected.

Stored voiceprints are neither “wave files” nor audio samplings, but instead are distinct computations of algorithmic information.

The key requirement is to identify the two sets of twins who are willing to participate in the test. Also determining the recording hardware and software platforms to use, choosing the apt voice collecting environment is the other important requirement to conduct the twin test. Two different recording hardware have been used for sample collection to add additional variation to the test results.

Prior to engage the twin candidates with the voice collection exercise, we have approached Ontario Tech university Research Ethics Board (REB) with detailed explanation of the voice collection exercise, and we asked for an exemption for a full REB approval. The reason for the exemption is that the objective of the test was a pure performance evaluation of the voice biometric products which is non-intrusive and no Personally identifiable information (PII) data of the candidates being collected during the exercise. After careful review of the request, the Research Ethics Board granted an exception approval for the voice collection exercise and each of the twin candidates were asked to sign the consent form (Appendix A), as per REB’s requirement.

The enrollment process and guidelines were provided by the vendor THC in detail, and the Identified candidates were briefed about the enrollment process in detail to ensure that they were aware of the expectations of the voice collection process. Candidates were asked to record their (fake) name, day/month and state/country of birth, favorite color as part of each enrollment process. Each candidate performed the enrollment twice, once as themselves and the other time as the imposter of their counterpart. The Voice collection tool provided by the vendor, THC, is capable of evaluating the quality of the samples in real-time and directs the candidate to repeat in case of poor quality.

All the recorded comparison scores from the verification process were statistically analyzed on a spreadsheet. The scores for each phrase are compared between the candidate score (A and B) and their respective impostor score (A’ and B’) per recording device for each pair of twins (Twins 1 and Twins 2).

The lowest score of the Candidate with their own rendition is colored in green and the maximum impostor score is colored in red. The average score has been calculated for each phrase to summarize the findings.

On analyzing the comparison scores generated by the VoiceVantage VoiceCheck™ tool, it is clearly evident that the possibility of successfully spoofing among the twins is not possible provided the phrases are exclusively specific to the candidate. The comparison score of the impostor went above 60% but below 75% only for phrases 2 and 3 which are “My voice is my password” and “city/state of birth” respectively using conventional recording devices. Over VoIP, the impostor’s scores are between 60% to 77% on average consistently.

In both cases, the phrases are factually ‘true’ for the impostor. On average the comparison score is fairly high when the impostor, no need to lie or to make an effort to impersonate his counterpart. This is clearly proving that voice is just not mere words, it carries the emotional characteristics of the person at the moment of recording. Voice tone and the difference in the psychological intent of telling the actual truth and trying to impersonate are really causing a huge difference in the voice samples.

The scores for the phrase ‘favorite color’ which is factually different for each of the twins and the impostor score for this phrase went absolute minimum when they must really try to spoof each other. This proves that choosing the phrase and its degree of exclusiveness to the individual is also a key factor in effective and efficient authentication.

The sensitivity of the voiceprints generated by VoiceVantage VoiceCheck™’s algorithmic technology to all those above-mentioned factors and properties of voice is very commendable and the results clearly prove that it is highly resistant to any such impostor attacks. Also, the performance of both the 32-bit and 64-bit tools are very similar.

We attribute the negligible differences in similarity scores between the two microphones to the hardware. We are certain that the collection environment and input quality were the same for both the devices. The test result shows that using the recording device with ideal settings/configuration will make a significant difference in determining the 'real' threshold and reduce the false acceptance rate.

Even though the nature of the tools is different (IOS app, web-based API tool), the impostor attacks were successful effortlessly on the other products. Considering the impostor attacks’ success rate on the products we tested and the range of similarity scores of the samples collected for VoiceVantage, it is clear that VoiceVantage is above par in efficient voice verification provided the phrases are uniquely related to the candidates.