Introduction

Since 2006, it has been known that widely arranged brain cortical response patterns elicited by individual face images with high-resolution functional magnetic resonance imaging (fMRI) can be used to discriminate between unique faces (1).  This work has been independently validated by other research laboratories (2,3). Face activation patterns obtained by fMRI are known to be related to and vary with the structure of the face (4) and these variations are consistent across individuals.

Cognitive engrams refer to multi-dimensional representations of brain activation in response to specific stimuli (1).  Cognitive engrams can be arranged into a [Rosetta] database which relates the Cognitive Engrams and other associated data to specific mental concepts, i.e., a visual representation of actual memory patterns.

Faces can also be analyzed and correlated with their physical features (5,6,7).  Relative sizes and distances for facial landmarks such as the eyes, nose, ears, chin, and skin texture, among others can be measured. Face data can be extracted from camera images, or from video streams. Principle methods for biometric face analysis are geometric, which is feature based, and photometric, which is view based. Many different algorithms for face analysis have been developed, including principal component analysis PCA, linear discriminant analysis LDA, elastic Bunch graph matching EBGM, and more recently deep-learning (DL) based non-linear feature extraction methods.

Face structure, as are all of our physical characteristics, are coded within DNA. Claes et al. (8) used extensive modeling methods to determine the relationships between facial variation and the effects of sex, genomic ancestry, and a subset of craniofacial candidate genes. Their modeling could lead to approximating the appearance of a face from genetic markers alone.

Knowing that face recognition by fMRI and by biometrics both depend on the physical differences between individual faces, a correlation of face recognition from these two different data sources was studied and reported herein. 

Methods

Overall, the steps used in this study are:

• Test subjects view pictures of face, object, or concept, or has other visual or auditory stimulation, while undergoing functional neuroimaging,
• Functional neuroimaging data is collected,
• 3-D Activation map is constructed, which constitutes the specific Cognitive Engram for the face / object imaged,
• Collection of activation maps is added to a (Rosetta) Database of activation maps
• The same faces used for generating fMRI activation maps are examined by (video) camera, and a biometric analysis is generated,
• One-to-one correspondence is made between the fMRI activation map (the facial cognitive engram) and the biometric data.

Preparation of Functional MR Imaging Activation Maps / Cognitive Engrams of Faces

3-D activation maps using fMRI can be prepared, as previously described (1). Briefly, normal volunteers are shown faces by rear projection screen or other methods (such as video projection goggles) while the test subjects are undergoing functional MR imaging. 

To perform fMRI, each test subject lays within a GE Cigna 3-T Signa 11X Excite MRI scanner, wearing a phased array whole head coil, mounted with a 45 degree mirror. This arrangement allowed test subjects to see images displayed onto a rear projection screen positioned by their feet. fMRI was performed while viewing the test stimuli it order to capture functional data, as described by Marks et al (1). A short localizer MRI scan was performed to verify that the field of view was within the skull, and to assure the absence of “ghost” images. A high-resolution full volume structural MRI scan was then obtained for each subject, using fast SPGR imaging (146, 1.0-mm thick axial slices, no spaces, TR = 8, TE = 3.2, FOV = 24 cm, 256 £ 256 matrix). These T1-weighted images provided detailed anatomical information for registration and 3-D normalization to a standard atlas.

Test subjects were then shown photos / images generated by PC using PowerPoint (Microsoft) and projected onto a rear projection screen placed at the foot of the test subject, as described in Marks et al (1). Pictures were viewed by means of a mirror system mounted on the head coil.

Changes in the blood oxygen level dependent (BOLD) MRI signal were measured using a gradient-echo echoplanar sequence. The following sequences were used, but variations are available. Continuous fMRI scans lasted 110 seconds each. EPI parameters were: TE 35, TR 2000, multiphase screen, 55 phases per location, interleaved, flip angle 90, delay after acquisition-minimum. Using a visual stimulus package, color photographs were presented in a mini-block design while neuroimaging was performed. In a typical session, after a 4 second lead-in time, a blank screen was displayed for 4 seconds, then the picture of interest for 4 seconds, and this was repeated for the scan time.

The fMRI scan volumes were motion-corrected and spatially smoothed in-plane. MRI data files were normalized and analyzed using MedX (version 3.4.3, Sensor Systems, Sterling, VA) to compute statistical contrasts and create a map representing significantly activated areas of the brain that responded differentially to a visual test stimuli.

For the voxels that show an overall increase in activity for meaningful stimuli, a positive regression analysis for the contrast between a test photo and control (blank page) stimuli was conducted, creating an activation map containing specific voxels with an uncorrected probability, P ≤ 0.05; meaning every voxel showing activation with the probability greater than 95%. Only those activated voxels were selected for further analysis. That statistical map was then superimposed on coplanar high-resolution structural images. The partial volume structural images were registered with the full volume high-resolution images using Automated Image Registration (9). Those full volume high-resolution images were then transformed (registered and normalized) to the Talairach and Tournoux atlas (10) using MedX tools. Each activated voxel on these images was selected to obtain Talairach coordinates of brain regions that respond maximally to the test stimuli and to further generate a Cognitive Engram. Comparison of observed patterns of activation were correlated with the nature of the response, such as face recognition, or a truthful or deceptive response

Three dimensional graphical representations of the identified activation maps were constructed by plotting the xyz coordinates, using the program DPlot (HydeSoft Computing, Vicksburg, MS). 

Preparation of Biometric Datasets of Faces

The same photographs of faces used to prepare fMRI data were then introduced to a biometric system. Ayonix Corporation (Tokyo, Japan) software was used, but other commonly available biometric systems should work as well. Biometric face data sets were then generated.

The Ayonix face extraction model uses customized HOG-like (Histogram of Oriented Gradients) features on a high number of overlapping face regions to reduce the effects of noise, viewing angle, aging, facial expressions and occlusions. This software extracts features from the whole face together, however, each feature window is aligned onto one of the facial landmark locations extracted in the pre-processing step. This way, the Ayonix software encodes both local and global information about the face geometry and appearance. The Ayonix software is constructed by training models on hundreds of thousands of faces from different age groups, genders and races, with different viewing and lighting conditions; thus appearance differences between different groups of people (ie. European and Asian) are inherently encoded in the feature extraction step of that software.

Skin color is not taken into account with the Ayonix software, and the images are converted to grayscale before processing. Following is a general outline of steps used to create face biometric data sets using Ayonix software:

Step 1. Face pre-processing

- Face is detected by face detection engine

- Face region is cropped and resized to a fixed image

- Face region is converted to grayscale

- Facial landmarks are extracted on the face

- Geometric alignment is performed

- Lighting effects are corrected

- Facial quality is measured

Step 2. Face feature extraction

- Face features are extracted from overlapped areas on detected facial landmarks

- Feature normalization and corrections are performed

- Features are transformed into recognition space using the Ayonix engine

Creation of Correspondence Grid:

A one-to-one correspondence grid was constructed (Figure 1 and Table 1). The two components were fMRI consensus activation points and biometric face data, arranged by each of the five static faces used in this study. Correspondence was made using regression analysis and other mathematical analyses.

Results

Using data from fMRI and biometrics, a one to one correspondence grid was constructed. A graphical illustration of the process is shown in Figure 1.  The correspondence grid is shown in Table 1.  The data for fMRI was previously provided (1).  The face biometric data sets are embedded into Figure 1.

Figure 1

Table  1 is a table of data in various formats illustrating how correlated data on brain activation obtained from functional MR imaging while viewing specific visual stimuli correlates with the biometric data obtained using facial recognition software.

Table 1

Table  1 is a table of data in various formats illustrating how correlated data on brain activation obtained from functional MR imaging while viewing specific visual stimuli correlates with the biometric data obtained using facial recognition software.

Discussion

A wide multivoxel activation pattern (1,17) is seen with fMRI during object and face recognition.  As noted in prior publications, and in an issued patent (18), these object-specific or concept-specific activations are referred to as Cognitive Engrams.  Cognitive Engrams may reflect neuronal population codes (1,17,18).  Marks et as and Muir (17) have previously shown (1,18) that Cognitive Engrams possess representational content. Various researchers, including Marks (1) and later by Kriegeskorte (2) and others have shown that fMRI activation data can be interpreted to identify individual, specific representational content, such as faces, unique objects, emotions, truthful and deceptive statements to questions, and other cognitive content

It is know that face recognition by biometric analysis (5) of a picture of a face is very dependent on the structure of the face.  Similarly, there is evidence that face representation by fMRI is also dependent on facial geometry. Loffler et al. (11) found evidence that neural activation patterns for individual faces are encoded as grouped data. This encoding varied on the direction (facial identity) and distance (distinctiveness) from standard or prototypical (mean) face. Loffler et al found that varying facial geometry (head shape, hair line, internal feature size and placement) caused the corresponding fMRI signal to increase with increasing distance from the mean face. Loffler also determined that the same neural population will respond to faces falling along single identity axes within this space. Boccia (12) found that the pattern of activity in most of these areas specifically codes for the spatial arrangement of the parts of the mental image

Rotshtein et al (13) found that fMRI of varying facial content demonstrated differential activity in critical face recognition areas of the brain.  The inferior occipital gyrus (IOG) showed sensitivity to physical rather than to identity changes.  The right fusiform gyrus (FFG) showed sensitivity to

identity rather than to physical changes. Bilateral anterior temporal regions show sensitivity to identity change that varies with the subjects' pre-experimental familiarity with the faces. These findings supported differential activity within the brain taking part in distinguishing varied facial content.
Rotshtein et al (4) used fMRI to study how the brain processes featural information and second  order spatial relations in face identity processing. Features included eyes, mouth, and nose. second-order spatial relations were measured between face features.  They found that feature-dependent effects occurred within the lateral occipital and right fusiform regions of the brain. Spatial relation effects occurred in the bilateral inferior occipital gyrus and right fusiform.  Overall, Rotshtein et al found that featural and second-order spatial relation aspects of faces make distinct contributions to behavioral discrimination and recognition. Face features contributed most to face discrimination, whereas second-order spatial relational aspects correlated best with recognition skills. These results support ongoing findings employing fMRI for face recognition tasks.

Rotshtein et al (14) then used "hybrid" faces containing superimposed low and high spatial frequency (SF) information from different identities. They found that repetition and attention affected partly overlapping occipitotemporal regions but did not interact. Changes of high SF  faces increased responses of the right inferior occipital gyrus (IOG) and left inferior temporal gyrus (ITG), with the latter response being also modulated additively by attention. In contrast, the bilateral middle occipital gyrus (MOG) responded to repetition and attention manipulations of low SF. A common effect of high and low SF repetition was observed in the right fusiform gyrus (FFG). Follow-up connectivity analyses suggested direct influence of the MOG (low SF), IOG, and ITG (high SF) on the FFG responses. Overall, their results showed that different regions within occipitotemporal cortex extract distinct visual cues at different SF ranges in faces and that the outputs from these separate processes project forward to the right FFG, where the different visual cues may converge. These results support ongoing findings employing fMRI for face recognition tasks and illustrate how analysis of differential brain activation demonstrates differential recognition of faces

Cohen et al (22) demonstrated that individual face images could be accurately reconstructed from distributed patterns of neural activity, even when excluding activity within occipital cortex.

Miyakawi et al (23) and Schoenmakers et al (24) showed that image reconstruction can occur based upon the brain activation pattern data alone, without the need for prior internal pattern references. Nishimoto et al (25) were able to interpret dynamic brain activity (viewing of movies) using a motion-energy encoding model and a Bayesian decoder.  

In essence, three dimensional activation patterns from wide areas of the brain are formed into patterns which are equated back to the stimulus for the activation (face, object, concept, emotion). Ultimately, all activation patterns in three dimensional space form unique data sets specific for the object or concept under consideration – Cognitive Engrams (1).

Analysis of faces by means of biometrics has its own complex art and science, as described elsewhere. Just as with visual recognition by the human eye, biometric systems depend heavily on differences in face structure, tone and other characteristics. There are a number of methods to analyze complex fMRI face activation data, including: sparse logistic regression (15), feature vectors (16) using a support vector machine algorithm; quantitative receptive-field models (3), multivoxel pattern information analysis (17), direction and distance from a prototypical (mean) face (11), partial least squares (19), and others. These methods are described in the referenced articles. 

Evidence for reverse lookup = mind-reading, a form of reverse lookup = mind-reading.

Cox (20) used multivariate statistical pattern recognition methods, including linear discriminant analysis and support vector machines, to interpret activation patterns of fMRI. Test subjects looked at categories of objects, rather than specific variations within a class. They were able to determine within some degree of experimental error which category (as opposed to unique individual) of object or picture their test subjects were looking at.

Thirion (21) used retinotopy of the visual cortex to infer the visual content of real or imaginary scenes from the brain activation patterns that they elicit. 

Yamashita (15) used a novel linear classification algorithm, called sparse logistic regression (SLR), to automatically select relevant voxels while estimating their weight parameters for classification. Using simulation data, they confirmed that SLR can automatically remove irrelevant voxels and thereby attain higher classification performance than other methods in the presence of many irrelevant voxels. These patterns of activatated voxels formed what can best be described as cognitive engrams, which can be used to predict or decode fMRI activity patterns. SLR also proved effective with real fMRI data obtained from two visual experiments, successfully identifying voxels in corresponding locations of visual cortex. SLR-selected voxels often led to better performance than those selected based on univariate statistics, by exploiting correlated noise among voxels to allow for better pattern separation.  

Extensive research in the published scientific literature, patent sources and internet indicate that a two-way lookup between fMRI and biometric data is a unique, original and not previously explored approach to face recognition.  

This concept further allows the practical interpretation of thoughts, emotions, feelings, intents using neuroimaging data. The reverse lookup concept allows the categorization and compiling of thoughts in the Rosetta Database, and ways to store and retrieve cognitive engrams. Correlation of individual biometrics to specific thought patterns via functional neuroimaging will further the identification of individuals and the interpretation of their associated concepts and intents.