Artificial Intelligence Driven Human Identification

Human Identification has proven to be a crucial part of numerous fields including authorization and forensics. With the advancements in technology, many open-source software and systems exist which can identify individuals with decent accuracy, but most of these systems rely on first-generation biometric modalities like facial recognition which are incompatible with the common CCTV cameras (recording in low-resolution) being used in almost every surveillance system. For the last few years, human identification using gait (i.e., an individual’s unique walking pattern/style) has been attracting all the researchers working in the fields such as surveillance and security, as gait-biometric is unobtrusive and can easily be extracted from low-resolution camera feeds. Existing systems working on gait analysis are yielding satisfactory results with some drawbacks, such as abnormalities in extraction (noisy silhouette) and exorbitant processing times.

The objective of this paper is to propose an automatic identification system that implements an artificially intelligent model which tracks and analyses an individual’s physical features while in motion from low-resolution camera feeds, keeping the speed-accuracy trade-off in mind. To achieve this, the low-resolution camera feeds are first pre-processed where the frame size is standardized and the clips are converted to grey scale, then using a series of image processing algorithms, the gait profile is extracted from the clips and saved in the form of numeric data. This data is then used to identify individuals using a trained AI (artificial intelligence) model.

The curiosity about gait as a biometric is strongly inspired by the want for an automated identification system for visible surveillance and tracking individuals. Currently, the deployment of gait as a biometric for identifying individuals in surveillance systems has attracted many researchers from the computer vision domain. The improvement in this research is being propelled by using improved extraction of features and optimized models. The suitability of gait recognition for surveillance systems emerges from the fact that gait can be perceived from a distance and that it is unobtrusive, i.e., non-invasive. Even though gait-biometric is still a brand-new concept, it overcomes almost all the issues that other biometrics suffer from including facial recognition, fingerprints, and iris recognition.

Understanding Gait and Biometrics

What is Gait?

The pattern in which an individual’s limbs move while in locomotion is known as Gait. It can be majorly categorized into two types: natural and trained, where natural gait is the style in which people instinctively walk and trained is learned through training. The gait of an individual can depend on many features including personality, sociocultural factors, age, and mood, and is immensely influenced by the nervous, musculoskeletal, and cardiorespiratory systems.

As (Khan, Jain, and Sharma, 2022) have shown, numerous features collectively form an individual’s gait profile, including step, step-time, step width, stride, cadence, speed, etc.

Gait consists of numerous parameters:

1. Step length – The distance between the point of initial contact of one foot and the point of initial contact of the opposing foot.
2. Stride length – The distance between successive points of first contact of the same foot.
3. Cadence – It is calculated in steps per minute.
4. Speed - The time it takes to walk a certain distance on the level ground over a short distance.
5. Dynamic base – It combines an inverted pendulum model of the stance leg with a pendulum model of the swing leg’s impact on the ground. The dynamic consequences of the heel strike at the end of each step might contribute to a periodic gait and passive stability.
6. Progression line – Between both left and right footprints, a line representing progression passes through the middle.
7. Foot angle – The angle formed by the line of progression and the foot axis is known as the foot angle.
8. Stride time – The time between successive heel touches of the same foot.
9. Stride width – The space between both the heels of the right and the left foot during a double stance.

Gait as a Biometric

Although gait analysis is the most recent biometric system, it is not a new concept in the field of study. Gait analysis has been explored and analyzed in numerous areas in the previous two decades. The Federal Bureau of Investigation (FBI) implemented gait analysis to narrow down their search on a list of suspects in the pipe-bomb investigation, on January 5th, 2021. In addition to biometrics, gait studies were pursued medical purposes, such as tracking relocations and as a screening tool.

The authors (Niyogi and Adelson,1994) presented the first gait identification system in 1994, which was built on a limited gait database. The Defense Advanced Research Projects Agency (DARPA) then created the famous HumanID initiative, which created the very first publicly available gait recognition database. Since then, several studies on gait recognition have been conducted. Most of the early gait recognition systems rely on video. Model-free and model-based techniques are the two types of video-based methodologies.

Advantages of Gait over other Biometric Modalities

Gait recognition has several characteristics, such as unobtrusiveness, that alternative biometrics don’t. These characteristics are briefly explained below:

1. Human gait can be caught from a distance. Other biometric systems, on the other hand, require that the subject be in close proximity to the biometric data collector or physically be in contact with it.
2. Low-resolution gait identification is possible, whereas other biometric technologies such as facial recognition, may not perform effectively with low-resolution clips/images.
3. Basic instrumentation can be used to recognize gait. Human gait can be collected using a camera, a mobile phone accelerometer, a floor sensor, or a radar.
4. Gait recognition can be performed without the person’s involvement. Other biometrics, such as fingerprints, require the user to place their finger on the sensor in order for the data to be collected.

Gait characteristics are difficult to imitate. This is because human silhouettes and activities are commonly used in gait recognition. This effectively means that it is almost impossible for an induvial to hide or change their own gait. This property is critical in criminal investigations.

Literature Review 

Table 1. Background work review

Methodology

To modelize the proposed approach, we used many techniques including data pre-processing, feature extraction, vectorization, etc. These methodologies are briefly explained below:

Dataset Collection

Several datasets exist such as the National Laboratory of Pattern Recognition dataset (NLPR), Carnegie Mellon University dataset (CMU), and Karadeniz Technical University dataset (KTU). But there is a common drawback to these datasets, that is, they are made with an emphasis on the physical features of the upper half of the human body. Additionally, many of such pre-existing datasets were limited in terms of noisy data and resolution. Thus, a custom dataset was required. A static camera position, to mimic the viewpoint of a CCTV camera (or any other surveillance camera) and plain background was used to set up our own data collection structure. 15 individuals walking laterally concerning the camera were recorded, where everyone was recorded 4 times with different view angles and camera positions. Out of the 60 video clips, 15 clips were used to generate the gait profile of each individual (i.e., the first clip for each individual generates their gait profile) while the remaining 3 clips for each individual are cross fed to generate the dataset. This cross-feeding approach works well when the problem is treated as a binary classification problem, and in turn, produces a larger training dataset

For example, out of the 15 individuals, if for individual X1, we recorded 4 walking captures V1, V2, V3, and V4, then V1 was used to generate the gait profile of X1, and V2, V3, and V4 were used to match the gait profile against the individual X1 as a positive match, and against some other individual X2 as a negative match. This way, a dataset of 15 individuals with four walking captures each can be used to make a dataset large enough to run training and tests.

Data Pre-processing

Getting the data ready for feature extraction is the first step toward gait analysis. The clips can be of different aspect ratios and frame sizes, this is generalized by setting the frame size to a standard value. Next, the clips are read frame by frame and each frame is simultaneously converted to greyscale so that the digital image processing algorithms can extract the features efficiently.

Figure 1. Screenshot of original video clip

Figure 2. Frame converted to greyscale

Feature Extraction

The resized greyscale frames are then used to extract the numeric gait data by running a sequence of algorithms on the frames. This whole extraction process can be broken down into steps:

Background Subtraction and Thresholding

This step is the foundation of this sequence of algorithms as every other step relies on the output of background subtraction for an efficient result. While several inbuilt functions are available, such as Open CV functions (BackgroundSubtractor, BackgroundSubtractorMOG2, and BackgroundSubtractorKNN), the proposed model implements median frame subtraction for a clean result. At random, 50 frames are selected from the clip which is then used to calculate the median frame. This median frame is used to obtain the absolute difference concerning each frame in the clip. The absolute difference is then blurred using median blurring. Further, adaptive thresholding is implemented to obtain a binary image with just the foreground being highlighted and all the rest of the information being dropped.

Figure 3. Median Frame used for Background Subtraction

Figure 4. Absolute difference between median frame and individual frame

Figure 5. Frame after median blurring

Figure 6. Binary foreground image obtained after thresholding

Region of Interest

To drop computational costs, and reduce time and space complexity, it is always preferred to process only the region of interest (ROI) rather than processing the whole frame. To obtain the ROI, a bounding box is placed using the contour of the human body, where firstly the whole frame is used to obtain a list of contours and then the largest contour (by area) is extracted and assumed to be that of the human body. Then a bounding box is placed on the contour, using the minimum and maximum values on the x and y axes of the largest contour.

This ROI boundary can also be used to filter the data extraction process to avoid extracting incomplete cycles. This is done by keeping a positive flag variable that gets a negative value as soon as the boundary of ROI touches the boundary of the frame. This flag variable is referenced when extracting the gait cycles and only the values where the flag variable was positive are computed.

Figure 7. Region of Interest.

Silhouette Outline Extraction

Once a clean region-of-interest, containing the human body silhouette, is obtained, the outline of the silhouette is extracted for further computation. But, before the outline is used for extracting the projections, the direction of motion of the human in the frame needs to be standardized, this is important as when the person is going from left to right, the projections tend to be more right-leaning (because of the natural leaning of a human in the direction of walk) and likewise the projections are left-leaning when the motion is in right to left direction. To filter this, the ROI is flipped when the motion is detected to be in the right-to-left direction, converting all the data for movement in the left-to-right direction.

Figure 8. Extracting silhouette outline from left-to-right movement

Figure 9. Extracting silhouette outline from right-to-left movement.

Top and Bottom Projections

The unidirectional outlines extracted in the previous step are then processed using a simple algorithm that returns a list of coordinates of the top and bottom projections of the silhouette. The top projection is extracted by calculating the distance between the top outline of the silhouette and the top border of the region of interest, i.e., the number of rows between the top border of ROI and the first pixel of the silhouette outline from above for each column. Similarly, the bottom projection is extracted by calculating the distance between the bottom outline of the silhouette and the bottom border of the ROI. Left and right projections don’t hold significant data when the subjects are walking laterally concerning the camera, hence are not included in this study. In Figure 10 below, the top projection is represented by red, and the bottom projection is represented by green.

Figure 10. Top Projections (red) and Bottom Projections (green).

Correlation

The correlation of consecutive frames is calculated to measure the randomness in motion. This data is the core data for a gait profile. To measure the randomness component, Pandas function corr() is used, which takes the current and the consecutive frame as inputs and calculates the correlation of the two images. Similarly, it takes all the other frames of the clip and calculates the correlation for each consecutive pair of frames. This process is repeated two times, one for each top and bottom projections. The results obtained in this stage are two vectors representing the randomness component of the top and bottom projections of the human body. These arrays are then normalized to fit the standard range and used to extract the periodic gait cycle.

Figure 11. Normalized Correlation of consecutive frames

Cycle Extraction

The human gait cycle is periodic. This property is very useful in terms of data collection. Generally, researchers split the complete gait cycle into two parts/half-cycles, this is practical as the first and the second half-cycles in a normal gait cycle are indistinguishable to a machine using only one camera, but this property can be further exploited to our benefit. The traditional method of cycle extraction describes the use of autocorrelation, but since we have one more periodic aspect in the frame (variations in the width of the ROI), the gait cycle can be divided into 4 parts/quarter-cycles (two quarter-cycles starting at the point where the ROI has a maximum width and ending with a minimum width and other two quarter-cycles starting with a minimum width and ending with maximum width). The complete gait cycle and the quarter cycles are depicted in Figure 12 below.

Figure 12. Gait Cycle.

Using the width of the ROI box, we can even filter out the incomplete gait cycles by marking the points where the subject being monitored leaves the frame. This is done by pointing out the abnormalities in the graph plot of the width of ROI (as when the subject leaves the frame, the width of the ROI jumps from the last value to the next value when he enters back in the frame, the smooth curve is broken, and a local maximum/minimum is achieved). This can be filtered easily. Figure 13 below shows the plots of autocorrelation and width of ROI, showing that the width plot captures the abnormalities of incomplete cycles much more efficiently than autocorrelation.

Figure 13. Width of ROI and Autocorrelation Plot.

The width plot is used to find all the maxima and minima, which are then processed to filter the incomplete cycles, Figure 15 below shows the quarter-cycles with the multi-colored lines, and the incomplete cycles are dropped where the line breaks.

Figure 14. Width plot highlighting the maxima and minima

Figure 15. Complete gait cycles (quarter-cycles) represented by the multi-colored lines

The projection vectors for the complete quarter-cycles are extracted and averaged, thus 4 curves are obtained, one for each projection, and this comprises of the gait profile that will be saved as a 4-feature value in the database.

Figure 16. Final Gait Profile of an individual (Correlation Projections)

Prediction

The gait profile extracted is saved in the database after converting it to numeric data. This data is read from the database where unique identification numbers are used to index the profiles. When a new video is analyzed, the ‘UID’ is required as an input, this number points to a gait profile saved in the database, against which the gait extracted from the test video is analyzed. Both the gait profiles are displayed for graphical comparison and finally a textual output with the prediction, “Positive”/”Negative” match label, and other metrics and information is displayed.

Binary Classification Approach

Since identification problems have multiple subjects to identify, hence these problems are usually categorized as multi-class problems. This approach is reasonable but not computationally very efficient, as it will automatically result in the dynamic number of classes (one new class for each person who is added to the database). This is avoided by using a binary classification approach, where the model is trained to recognize the difference between a positive match and a negative match. Hence, there are only two classes, 1 for positive matches and 0 for negative matches.

Results

The model can successfully identify individuals using gait profiles, which are extracted from low-resolution video clips, and it can do so with remarkable speeds (in a range of 20 to 35 Hz depending on the hardware specifications of the machine).

Let’s consider two test cases, first test case A where the person in the test video is the same person whose ‘UID’ is entered for the database profile, and another test case B where the person is the test video is not the same person and is being analyzed against someone else’s gait profile. The outputs for both test cases are given below in Figure 17-20.

Figure 17. Graphical Output for Test Case A

Figure 18. Metrics and Final Predictions for Test Case A

Figure 19. Graphical Output for Test Case B

Figure 20. Metrics and Final Predictions for Test Case B

In the test cases, textual outputs in the figures, Figure 17, and Figure 19 shows that the average processing speed was 34 FPS when the Nvidia CUDA toolkit was used and 22 FPS when standard toolkit and processing systems were used.

Further, this proposed artificial neural network was compared with other machine learning models (including logistic regressor, decision tree classifier and random forest classifier) and their Receiver Operating Characteristic (ROC) curves are plot and the area under the curves are calculated. Figure 21 below shows the ROC plots, and it is visible that although the proposed model outperforms the other models, machine learning models are also performing decently as the input features were extracted efficiently and is a numeric fashion.

Figure 21. ROC Plot

Conclusion

The objective of the paper, i.e., “Human Identification using Gait Analysis and AI” is achieved, and an automatic identification system based on gait analysis is proposed. The applications of this research in the domain of security, surveillance, and authorization are studied and a conclusion is drawn that gait as biometric is not only feasible, but it is also the next practical upgrade in the surveillance and security domain.

The existing models capable of analyzing gait achieve decent accuracies but at the cost of high processing times. The proposed model maintains the speed/accuracy trade-off, where it processes the frames at a speed of 20-35 Hz and achieves more than 85% accuracy in average cases, with as high as 99% accuracy in ideal cases.

The major breakthroughs in the proposed model are achieved based on two optimizations, firstly implementing algorithm optimizations, and using multi-threading, and secondly dividing the gait cycle into 4 quarter-cycles rather than 2 half-cycles. This facilitates efficient gait extraction and most of the data is utilized, without data loss.

Conflict of interest

The authors declare no potential conflict of interest regarding the publication of this work. In addition, the ethical issues including plagiarism, informed consent, misconduct, data fabrication and, or falsification, double publication and, or submission, and redundancy have been completely witnessed by the authors.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.