US 5,148,477 AGrant
Method and Apparatus for Detecting and Quantifying Motion of a Body Part
Issue Date:1992-09-15
•62 Claims
•3 Drawing Sheets
Abstract
A method for detecting and quantifying motion of a body part of an individual or animal, such as an individual's face or portion thereof. A digitized image of the body part in a reference position is produced (referenced image). A digitized image of the body part in a changed position then is produced (changed image). The referenced image is compared with the changed image to determine movement parameters indicating the movement of the body part from the referenced image to the changed image.
Metadata
Assignee
- Board of Regents of the University of Oklahoma
Inventors
- John G. H. Neely
- John Y. Cheung
Application Information
Application Number:US 5735025
Filing Date:1990-08-24
Priority Date:1990-08-24
Art Unit:266
Classifications
IPC:
G06K 900
Field of Search:
3823583646;42;48;44;1;18;51105413.13;413.16;413.22;413.23;413.27;413.02
Patent Drawings (3 sheets)
Description
Field of the Invention
The present invention relates generally to methods for detecting and quantifying motion of a body part, and more particularly, but not by way of limitation, to a method for detecting and quantifying motion of a body wherein a digitized image of a body part in a control position is produced and a digitized image of the body part in a changed position is produced and the two images are compared to determine movement of the body part between the two digitized images.
Brief Description of the Drawings
FIG. 1 is a diagrammatic, schematic view of the system of the present invention.
FIG. 2 is a flow chart diagrammatic illustrating the program used in connection with the processor shown in FIG. 1.
FIG. 3 is a diagrammatic view of an individual's or patient's face to be analyzed using the system of FIG. 1, and showing grid lines and areas of interest used in the system of FIG. 1 and showing such grid lines and areas of interest as they would appear on the screen of the computer monitor portion of the system shown in FIG. 1.
FIG. 4 is a diagrammatic view of the face of FIG. 3 with marker dots placed on the face for use in aligning frames to be compared which are used in one embodiment of the present invention.
FIG. 5 is a pictorial view of the face of FIGS. 3 and 4 showing the individual's head located in a head holder.
FIG. 6 is a side view of an individual with its head disposed in the head holder.
FIG. 7 is a partial perspective view of the head holder of FIGS. 5 and 6.
Description of the Preferred Embodiments
Treatment of paralysis, such as facial paralysis, has been confounded by lack of ability to measure in a quantitative fashion the degrees of paralysis or paresis, recovery, and abnormal movements such as synkinesis. The major difficulty in measuring movement is the complexity of the neural, muscular, and ultimately the biomechanical structure and characteristic of the skin and the underlined musculoaponeurotic system, thereby creating not only linear and curvilinear movements, but also the three dimensional deformations along the lines of maximum tension.
At present, the standardized technique for quantifying degrees of facial movement consist of the six point clinical grading scale proposed by John House, M.D. and adopted by the American Academy of Otolaryngology-Head and Neck Surgery. This grading scale is subjective, some what arbitrary and overlapping. There is no grading scale, clinical or otherwise, for synkinesis.
The system of the present invention provides a means for detecting and quantifying body parts and, more particularly, facial movements (shape changes over a defined surface area).
Using the motion analysis apparatus 10, movements such as facial and body movements, easily can be detected and quantified. Gradations in effort can be recognized and quantified. The variations in observer appreciated movement of the individual's face versus the pixel quantification between sides of the face and between maximum and minimum movement are valid. It has been found that test/retest measurement utilizing the same cursor designated area and same degree of maximal movement have been shown to be quite reliable. The results of the system described herein are applicable for detecting appreciable, under-appreciated and even unappreciated movements.
In some facial movement, such as the maximal movement of the nose superiorly as though one were sniffing a particularly unpleasant odor, not only is the expected nears movement noted, but additional accessory strategies which vary considerably among individuals may be observed. The present computer analysis system permits the separation of the accessory strategy movement from the requested movement and the analysis of these two movements in combination or separately.
The present system can separate brow movement into at least three components (forehead wrinkling, eyebrow movement and upper eyelid movement). The present system also can analyze these movements individually or collectively.
The system of the present invention can identify three components of eye closure (upper eye lid movement, lower eye lid movement, and surrounding orbicularis oculi movement) and analyze these components individually or collective.
The system of the present invention also can detect several components of smiling at cites around the mouth and distant from it. For example, movement of the commissure, movement of the nasolabial fold, movement of the eye and upper cheek, movement along the distribution of the zygomaticus major.
The system of the present invention can identify and measure movement in secondary areas to the primary area of requested movement thereby allowing quantitative measurement of abnormal facial movement, such as synkinesis.
It should be noted that, although the system of the present invention has been described herein, in some instances, with specific reference to detecting and quantifying facial movements, the system of the present invention is useable for detecting and quantifying motion of any body part. The references herein to facial movements are not to be construed as limiting the present invention to facial movement. The present invention also is useful for analyzing motion of a body part of an animal.
Shown in FIG. 1 is a diagrammatic, schematic view of a motion analysis apparatus 10 which is constructed in accordance with the present invention. The motion analysis apparatus 10 includes an image producing means 12 which is constructed and adapted to produce a digitized image of the body part and to output the digitized image over a signal path 14. The digitized image outputted over the signal path 14 is inputted into a processor 16. The processor 16 processes the digitized image in a manner to be described in greater detail herein.
A computer monitor 18 is connected to the processor 16 via a signal path 20. The computer monitor 18 includes a keyboard 22. A printer 24 is connected to the processor 16 via a signal path 26. The processor 16 is adapted to output images in a printer readable format over the signal path 26 and the printer 24 is adapted to output the images received from the processor 16 in a human perceivable printed format.
A display monitor 28 also is connected to the processor 16 for receiving an image from the processor 16 via a signal path 30 and for displaying the received image.
The image producing means 12, more particularly, comprises a video camera 32. The video camera 32 responds to the intensity of light reflected off an image. The light intensity is converted into an analog signal by the video camera 32. The brighter the light intensity, the higher the amplitude of the analog signal. The video camera 32 produces the video images in the analog format and outputs the video images in the analog format over a signal path 34.
The video images in the analog format are inputted into a video cassette recorder 36. The video cassette recorder 36 records on a video medium the video images in the analog format received from the video camera 32 on a video tape.
The video images in an analog format are outputted by the video cassette recorder 32 over a signal path 38. The video images outputted by the video cassette recorder 36 are inputted into a frame grabber 40. The video images outputted by the video camera 32 and the video cassette recorder 36 consist of a number of individual frames. The frame grabber 40 includes a high speed analog to digital converter and a high speed digital to analog converter. The frame grabber 40 is constructed to capture a single frame of the video images outputted by the video cassette recorder 36 in the analog format and to digitize each scan line in the captured frame to produce a digitized image which is outputted over the signal path 14. In the frame grabber 40 each scan line is sampled at a constant rate. The amplitude of the analog signal which corresponds to the light intensity at that spot of the image now is represented by a digital number. The more light intensity there is at a given spot of the image, the higher the digital number. Each spot is called a pixel (image element) and the corresponding digital number is the value of that pixel.
In the alternative, the video images in the analog format are outputted via the video camera 32 over the signal path 34 and over a signal path 42 whereby the video images in the analog format are directly inputted into the frame grabber 40. In this last mentioned instance, the video cassette recorder 36 is by-passed and a video tape record is not necessarily maintained of the video images outputted by the video camera 32. In either instance, the frame grabber 40 digitizes the video images in the analog format and outputs the digitized images over the signal path 14.
In the alternative, the body part to be analyzed is recorded via a camera 44 which is adapted to produce a still image or frame in a digitized format of the photographed body part and to output the still frame in a digital format, a digitized image. Thus, the camera 44 outputs the digitized image over a signal path 46.
The processor 16 is connected a disk storage 48. The digitized images outputted by the camera 44 over the signal path 46 or the digitized images outputted by the frame grabber 40 over the signal path 14 can be inputted into the disk storage 48 over a signal path 50 and stored therein as directed by the processor 16.
In general, an individual is placed in front of the video camera 32 and the video camera 32 is focused on the body part to be analyzed. The body part to be analyzed is positioned in a reference or control position. The individual then moves the body part to be analyzed to a changed position. It should be noted that it may be desirable to move the body part to be analyzed from the control position to the changed position and back to the control position in some applications. Also, it may be desirable to have the individual repeat the movement a number of times so the operator will have a number of images from which to select the desired images for processing.
The video camera 32 produces the video images in the analog format of the body part to be analyzed through the movement of the body part from the reference position to the changed position. These video images outputted by the camera 32 are recorded on video tape by the recorder 36 or directly inputted into the frame grabber 40. If recorded on tape by the video cassette recorder 36, the recorded video images then are played back and outputted by the video cassette recorder 36 and inputted into the frame grabber 40.
The video images in the analog format are outputted from the frame grabber 40 via the signal path 30 to the video monitor 28. Each frame of the video images also are digitized to produce a series of digitized images and the digitized images are outputted from the frame grabber 40 over the signal paths 14 and 50 and inputted in disk storage 48, as directed by the processor 16.
When it is desired to analyze the images, the digitized images are outputted from disk storage 48 over the signal path 50 and inputted into the frame grabber 40, as directed by the processor 16. The frame grabber 40 converts the digitized images to video images in the analog format. The video images in the analog format are outputted over the signal path 30 and inputted into the video monitor 28 where the video images are displayed. More particularly, the frame grabber 40 and the processor 16 are adapted to sequence through the digitized images frame by frame and each frame sequentially is displayed on the video monitor 28. The operator using the keyboard 22 instructs the processor 16 to cause the frames sequentially to be displayed.
In this capture mode of operation, the operator can select either one of two modes of operation: a select pair mode or a select sequence mode. In the select pair mode, the operator sequences through the images, while viewing each image on the video monitor 28, until the operator is satisfied that a particular frame represents a good image of the body part in the control position. The operator then operates the keyboard 22 to instruct the processor 16 to output a capture code over the signal path 14 which is inputted into the frame grabber 40. In response to receiving the capture code, the frame grabber 40 captures the frame and this captured frame in a digitized format is outputted by the frame grabber 40 is marked and stored in the processor 16 memory or inputted into the disk storage 48 where it is identified as a digitized control frame or digitized control image.
In the select pair mode, the operator then continues to sequence through the frames until the operator is satisfied that a particular frame represents a good image of the body part in the changed position. The operator then causes the capture code to be outputted by the processor 16. In response, the frame grabber 40 outputs the captured frame in the digitized frame which is stored in the processor 16 memory or inputted into the disk storage 48 where it is identified as a digitized moved frame or digitized changed image.
The operator then instructs the processor 16 by way of the keyboard 22 to process the selected control frame and the selected changed frame. In the process mode of operation, the processor 16 causes the digitized control image and the digitized changed image to be withdrawn from disk storage 48 into the processor 16 memory if they are not already previously stored in the processor 16 memory. The processor 16 then compares the digitized control image to the digitized changed image and the processor 16 produces a digitized compare frame or digitized compare image which is indicative of the movement of the body part from the control frame or image to the moved or changed frame of image. The processor 16 is programmed to determine movement parameters indicative of the movement from the control image relative to the changed frame or image and to output the movement parameters, as well as to cause the digitized compare image to be processed through frame grabber 40 which outputs the compare image in the analog format. The compare image in the analog format is outputted over the signal path 30 for display on the video monitor 28.
In addition, the determined movement parameters are outputted over the signal paths 14 and 20. The determined movement parameters are displayed on the computer monitor 18 screen. The processor 16 also can cause the determined movement parameters, as well as the compare image, to be outputted on the signal path 26 for outputting the compare image and the determined movement parameters in the printed format via the printer 24.
As mentioned before, the operator also can select the select sequence mode of operation, rather than the select pair mode of operation just described. In the select sequence mode of operation, the video images are produced and each frame of the video images is digitized via the frame grabber 40 and stored in the processor 16 memory or disk storage 48 in a manner exactly like that described before with respect to the select pair mode of operation.
In the select sequence mode of operation, the operator sequences through the images while viewing each image on the video monitor 28. Using the keyboard 22, the operator selects a particular frame which represents a good image of the body part in the control position and the operator instructs the processor 16 to mark the selected frame.
After selecting the control frame, the operator then sequences through the frames and selects a plurality of moved or changed frames. For example, the operator may select a plurality of moved or changed frames between the control frame showing the body part in the control position through the frames showing the body part in the changed position, and back through a frame showing the body part in a position similar to the control position.
The control frame and the moved or changed frames each are assigned a number. In one present embodiment of the invention, up to thirty-six frames can be selected as the control frame and the moved or changed frames. The number of frames that can be selected is limited only by the availability of the processor 16 memory. The selected control frame and each of the selected moved or changed frames are stored in the disk storage 48 along with a reference number identifying each of the selected frames.
The operator then instructs the processor 16 by way of the keyboard 22 to process the selected frames. The processor 16 is conditioned in the process mode of operation in response to this command.
In the process mode of operation, processor 16 causes the digitized control frame and the next selected moved frame to be withdrawn from disk storage 48 into the processor 16 memory if they are not previously stored there already. The processor 16 then compares the digitized control image or frame to the digitized changed image or frame and the processor 16 produces a first digitized compare frame or first digitized compare image which is indicative of movement of the body part from the control frame or image to the moved or changed frame or image. The processor 16 also is programmed to determine movement parameters indicative of the movement change from the control frame or image relative to the moved or changed frame or image.
The processor 16 then compares in sequence each of the moved or changed frames or images with the control frame or image to produce a plurality of digitized compare frames or images, each of which is indicative of the movement of the body part between the control frame or image and each respective moved or changed frame. The processor 16 is programmed to determined movement parameters indicative of the movement change from the control frame or image relative to each of the moved frames or images. The movement parameters determined by the processor 16 are outputted by the processor 16 over the signal path 20 and displayed on the screen of the computer monitor 18. In addition, the movement parameters indicative of movement change can be outputted over the signal path 26 and printed by the printer 24 to provide a list of the determined parameters in a printed format.
Video cameras such as the video camera 32, video cassette recorders such as the video cassette recorder 36 and cameras such as the camera 44 are all well known in the art and a detailed description of the construction and operation of these components is not deemed necessary. The format of the video tape used in the video cassette recorder 36 can be Beta, VHS, PAL, or 1/2" or 3/4" video tape or any other suitable video tape format. Also it should be noted that a video disk recorder can be used in lieu of the video cassette recorder.
Frame grabbers which are constructed to operate in a manner described herein with respect to the frame grabber 40 are well known in the art and a detailed description of the construction and operation of such frame grabbers is not deemed necessary. For example, one frame grabber suitable for use with the present invention is a frame grabber sold by Data Translation, and identified by Model DT-2853. This frame grabber is constructed to receive video images and to receive capture commands and to isolate and capture a selected frame in the video images received by the frame grabber as instructed by the received capture command. This commercially available frame grabber also is adapted to digitize each scan line in the selected or capture frame. This type of frame grabber also is constructed to receive digitized images and convert such digitized images to their analog formats for viewing in human perceivable forms such as via the display monitor 28.
A processor such as the general purpose digital computer available from Master Systems, Model Executive AT has been found suitable for use as the processor 16 of the present invention. This processor was used in connection with a disk storage commercially available Seagate, Model 225, and this system was found suitable for use in the present invention.
It should be noted that there are available other types of frame grabbers which can be used to combine the digital image with messages from the processor 16 using a digital video interactive (DVI) protocol. In this case, only one high resolution video monitor is required. Further, in lieu of the keyboard 22 used with the computer monitor 18, an optional interface box with push buttons can be used to communicate with the processor 16.
To detect motion, the subregions of the same image that have been moved need to be identified. This requires at least two images. The first one is referred to herein as the control image or control frame. The second image is the changed image where subregions of the body part have been relocated referred to herein as the moved image or moved frame. In the case of facial motion analysis, the control image can be the face at rest while the second or moved image can be the face after a specific feature, such as the eyebrow, has been moved. Alternatively, the comparison also can be made between the left half and the right half of the same image or frame.
In the processing mode of operation, the processor 16 performs four distinct processes: (1) image pre-processing; (2) motion detection, (3) image post processing and (4) measurement.
Image Pre-Processing
In the image pre-processing step, digitized images are processed to yield a consistent image quality. This is an optional step. Due to differences in lighting between images and between the various features of the image, pre-processing is necessary to equalize or normalize the pixel ranges of the digitized images or frames. The lighting can be changed from one image to another at the same photo session or between series of images. The equalization process compensates for the varying intensity and minimizes the shadow zones often occurring from three dimensional objects. In the system of the present invention, the equalization process is accomplished by equalizing the histogram of pixel values into a specific distribution. This equalization step produces images that have an average intensity throughout and with a specific distribution of pixel intensity variations.
The captured images are usually taken under varying lighting conditions dependent on the environmental setting. Even when the setting is carefully controlled, the light reflected from the face also may vary according to the complexion, skin color, head shape, skin texture and other facial features. Therefore, it is essential to normalize the intensity of the image. From the existing intensity histogram, the pixel levels are remapped to give a specific histogram shape such as a bell curve, a Poisson curve, a linear curve, a logarithmic curve, or a user defined curve. In the present invention, a bell curve was utilized so that the equalized images had the same average intensity and the same intensity deviation for later comparisons between two images or between sets of images.
Motion Detection
After pre-processing the images or frames, the subregions of the image or frame that are different between images are highlighted. This is done by comparing two images or frames and removing areas that are common in the two images or frames. When the common or subregions are suppressed, only the regions that have been moved remain. This is the main operation for detecting motion. By examining the two images, the common area can be subtracted out thus leaving a small residual pixel value. In an idea case, if the two images were identical, there would be no residual value for all pixels. If any subregion in the image have been relocated, a substantial difference in pixel value results after the subtraction. The relocated region can now readily be differentiated by their pixel values from the other common regions.
After the subtraction of the two images, negative values may arise. Three methods are possible for handling such negative values: (1) negative values are treated as zero; (2) negative values are changed to absolute values; and (3) all values are scaled and offset to eliminate negative values. When negative values are treated as zero, most moved regions are highlighted. When negative values are changed to absolute values, a true difference between two images or frames is obtained. This last mentioned scheme may provide more accurate assessment of the moved regions. When all values are scaled and offset to eliminate the negative values, the difference in images or frames seen may be advantageous for some type of measurement. In the system of the present invention, negative values were changed to absolute values.
Image Post Processing
After the common areas between images have been removed, the last step is image post processing. This is an optional step. In this step, an image enhancement process first is utilized to bring out the regions that have been moved. Then, a digital filter in software may be used to clean up unwanted and spurious spots in the image.
For image enhancement, it is desirable, after image subtraction, to highlight the moved regions for visual study and verification. The subtracted image may be enhanced through a number of techniques: (1) contrast enhancement, or (2) thresholding. In contrast enhancement, the subtracted image is enhanced so that the brighter regions appear to be more bright. This essentially is a histogram equalization process. In thresholding, the gray scale is divided so that the pixels above a certain threshold appear extremely bright, while those pixels below the threshold appear extremely dark. In other words, the image is thresholded so that a small residual errors in the common regions are replaced by a null value, while those pixels representing the dislocated or moved regions that are above the threshold are amplified to the maximum values. The common regions have zero pixel values and the dislocated or moved regions have maximum pixel values. The threshold can be set manually or automatically. In one embodiment of the present invention, thresholding was utilized in the image post processing step and the threshold values were set manually by the operator using the keyboard 22.
It is not unusual that, after the image subtraction process and the image enhancement process, many spurious points may remain in the image. The spurious points are usually isolated points placed randomly around the image. Therefore, a further step in post processing is to remove these isolated, spurious points. This may be done by a cleansing filter such as a Sobel filter that sharpens the enhanced regions and deemphasizes the isolated points. At this stage of the process, the remaining image or frame is ready for quantitative measurements.
Measurement
The final step is to quantify (determine the movement parameters) the regions of interest and to measure the movement in terms of the brightness of the subtracted image. This quantification step can be accomplished in a number of different ways. Two possible approaches are pixel counting and area integration. In pixel counting, the area of movement is measured by the number of pixel above a certain threshold. Therefore, the moved region is approximated by counting the number of pixels above the threshold. In area integration, the moved region is approximated by the summation of the actual subtracted values, i.e., the difference of the two images in gray level. The last mentioned approach may be more indicative of the movement and possibly the effort of movement. In one operational embodiment of the present invention, pixel counting was used to quantify the regions of interest and to measure the movement in terms of the brightness of the subtracted image.
Shown in FIG. 2 is a diagrammatic view of the software or processing operations performed via the processor 16. In other words, the processor 16 is programmed to operate in this manner.
The program in the processor 16 first is invoked and initialized (START 52 in FIG. 2). In response to receiving a capture code inputted into the processor 16 via the computer keyboard 22 by the operator, the processor 16 is conditioned in the capture mode of operation (Image Capture 54 in FIG. 2).
In the capture mode of operation, the digitized images are captured and transferred to the processor memoy and optionally the disk storage 48.
After the digitized images have been captured, the captured images can be previewed on the display monitor 28 (Image Preview 56 in FIG. 2) to insure that the digitized images of interest have in fact been captured and stored.
After previewing the digitized images, the operator can select the particular images to be stored for later selection and processing (Image Selection 58 in FIG. 2). It should be noted that during the image preview 56 or the image selection 58, the digitized images are reconverted into images visually perceivable in a human viewable format and outputted on the video monitor 28. The operator or user then uses the keyboard 22 of the computer monitor 18 to effect the selection of images being displayed on the display monitor 28. The selected digitized images are stored in the disk storage 48 as controlled by the processor 16 (Stored Image 60 in FIG. 2) or the digitized images may be inputted directly from the camera 44 (Digital Still 62 in FIG. 2).
In any event, the selected digitized control image and the selected digitized moved or changed image are taken either from the disk storage 48 where the images previously have been stored or from the frame grabber 40 where the images were just captured or from the digitized still camera 44. The images to be analyzed by the processor 16 are selected by the user and the user marks the areas of interest in the selected digitized control image and the selected digitized changed image. The digitized control image and the digitized changed image then are processed via the processor 16.
In the process mode of the processor 16, the selected images (a selected digitized control image and the selected digitized changed images) are screened and analyzed to remove various artifacts. The purpose of this step is to eliminate white spots and to account for any differences in skin pigmentation, uneven lighting reflection and undesirable image translation. The primary tool used to accomplish this step is histogram equalization indicated in FIG. 2 by the reference numeral 64.
After histogram equalization 64, the digitized control image and the digitized changed image are compared through a subtraction process (Subtraction 66 in FIG. 2) or a difference process (Different 68) or a biased.. subtraction process (Bias Subtraction 70) to produce a digitized compare image. The digitized compare image then is processed to remove artifacts and the moved regions in the digitized compare image are enhanced. This step is accomplished by way of histogram equalization 72.
After histogram equalization 72, the digitized compare image is processed to quantify the movement either through the counting of threshold pixels (Threshold 74 in FIG. 2) or through integration of pixel gray values in the area of interest (Gray Value 76 in FIG. 2). After the measurement step, the processor 16 software returns to the main menu for further processing (END 75 in FIG. 2).
It should be noted that at any step in the image capture or processing or detection or post processing or measurement steps, the operator has the option to return to the main menu.
Set forth below is the program which was used to operate the processor 16 in one operational embodiment of the present invention. ##SPC1##
In the specific software set forth above, a logo screen initially was invoked and then a main menu was presented. The main menu in this case had four primary modes: the capture mode; the load and store mode; the process mode and the configuration. When the capture mode was selected, another screen was presented that solicited from the user whether or not the user wanted to proceed or change parameters. If the change parameter submode was selected, then the system parameters related to image capture were made available for the user to change and modify. Some typical parameters at this point were normal frame rate, slow frame rate, number of frames for example. If the user selected to proceed with the capture mode, another screen called the capture selection screen was presented where the user could specify the normal rate or slow rate. After the selection of capture rate, another screen was presented to the user where the capture process could be initiated. After the capture process was completed, the user could select to view the captured images individually or as a whole on the display monitor 28. If the user chose the load and store mode, several options were available. These options included retrieving from the disk storage 48, saving to the disk storage 48 and viewing the current images individually or as a whole on the display monitor 28. After the images were viewed via the display monitor 28, the user then was asked to select frames for processing. The user could pass through the frames using cursor keys and marking the frames designated for later analysis. The frames could be marked in two ways. First, pairs could be selected individually for analysis or, alternately, a series of frames could be selected for comparison with a particular reference frame. After the frames were selected, the system automatically returned to the main menu where the user could select the process mode.
In the process mode, screens were presented to the user as an example of the procedure. The processing of the images began with a normalization procedure of the control image, then the subtraction step between the control image and the changed image is effected. At this point, the user was asked to define the coordinates of the body part or face. This was done without the use of marker dots or with the use of marker dots on the face or marker dots on the head holder if available. This step may be done automatically if maker dots were present. After the coordinates were defined, then the user was asked to define the areas of interest using cursor keys. After the areas of interest were defined, the system then proceeded to measure new regions within the defined areas of interest. For random pairs, the system proceeded in the same manner, after the definition of coordinates in the boxes, the system could enter a batch mode and process all of the images at the time. The output of the measurement was recorded on a file for storage or for later printout. After processing, the system return to the main menu for further instruction.
As mentioned before, the present invention particularly is adapted to detect and analyze facial movement.
Diagrammatically shown in FIG. 3 is a patient 80 or, more particularly, a patient's face. Initially, marker dots 82, 84, 86 and 88 are placed on the patient's face, as shown in FIGS. 4 and 5. The mark dots 82, 84, 86 and 88 provide a means to align images or frames. Also, shown in FIGS. 5, 6 and 7 is a head holder 90, and the marker dots or at least three marker dots could be placed on portions of the head holder 90, rather than on the patient's face.
It should be noted that other means also could be utilized to align images or frames for processing. Also, it should be noted that, in one embodiment of the invention, marker dots were not utilized. If marker dots are not utilized, it is important that the patient's 80 head be secured in the same position between images and the camera 32 be positioned in exactly the same position between images with exactly the same camera 32 settings. The head holder 90 comprises a holder stand 92 with only the upper portion of the holder stand 92 being shown in FIGS. 6 and 7. The holder stand 92 may be adapted to be supported on a floor surface or, in the alternative, the holder stand 92 may be adapted to be mounted or supported on a wall surface. The holder stand 92 preferably is cylindrically shaped, although the holder stand 92 could have any other shape if desired in a particular application.
The head holder 90 includes a top pad assembly 94 and a chin pad assembly 96. The top pad assembly 94 is movably connected to the holder stand 92 for movement in a vertically upwardly direction 98 (FIG. 6) and in a vertically downwardly direction 100 (FIG. 6).
The top pad assembly 94 comprises a top bar 102, a top sleeve 104 and a pad 105. The top sleeve 104 has an opening (not shown) extending therethrough and the holder stand 92 is slidingly disposed through the opening (not shown) in the top sleeve 104. One end of the top bar 102 is connected to the top sleeve 104 and the pad 105 is connected to the opposite end of the top bar 102.
The chin pad assembly 96 comprises a U-shaped chin bar 106, a chin sleeve 108 and a pair of chin pads 110 and 112. The chin sleeve 104 has an opening (not shown) extending therethrough and the holder stand 92 is slidingly disposed through the opening (not shown) in the chin sleeve 108. A mid-portion of the chin bar 106 is connected to the chin sleeve 108. One of the chin pads 110 is connected to one end of the chin bar 106 and the other chin pad 112 is connected to the opposite end of the chin bar 106.
In operation, the patient 80 is seated in front of the head holder 90 with the patient's 80 head disposed generally between the top pad assembly 94 and the chin paid assembly 96. The top pad assembly 94 is moved in the downwardly direction 100 until the pad 105 engages a central portion of the top of the patient's 80 head. The top pad assembly 94 is secured in this position via a fastener 114 which extends through the top sleeve 104 and engages the holder stand 92 to secure the top sleeve 104 in the one adjusted position on the holder stand 92. Then, the chin pad assembly 96 is moved in the upwardly direction 98 to a position wherein the pad 110 engages one side of the patient's 80 chin and the pad 112 engages the opposite side of the patient's 80 chin. The chin pad assembly 96 is secured in this position via a fastener 116 which extends through the chin sleeve 108 and engages the holder stand 92 to secure the chin sleeve 108 in the one adjusted position on the holder stand 92. The patient's 80 head is held in this stationary position secured between the top pad assembly 94 and the chin pad assembly 96.
The video camera 32 is positioned in front of the patient 80. The video camera 32 is focused on the patient's 80 face with the patient's 80 face occupying virtually or substantially the entire image seen by the video camera 32.
It is important to adjust or create the proper lighting about the patient's 80 face. In one embodiment of the present invention, it has been found that soft, low power lighting which comes in from both sides of the patient's 80 face, rather than harsh, studio bright lighting, yields the best results.
With the patient's 80 head secured in the head holder 90, the video camera 32, the video cassette recorder 36 and the video monitor 28 are turned on. The video cassette recorder 36 is conditioned in the record mode and the video camera 32 also is conditioned in the record mode. The video images in the analog format outputted by the video camera 32 over the signal path 34 are inputted into the video cassette recorder 36 which records the video images on video tape. In addition, the video images are outputted over the signal paths 42 and 38 through the frame grabber 40 which outputs the video images over the signal path 30. The video images outputted over the signal path 30 are inputted in the video monitor 28 which displays the received video images. In this manner, the operator is able to view the images being recorded via the video cassette recorder 36 to be sure that proper or desired images in fact are being recorded.
When using the motion analysis apparatus 10 for analyzing facial movements, the operator, for example, would request the patient 80 to perform four tasks: (1) raise eyebrows, (2) close eyes tightly, (3) squint or wrinkle nose, and (4) smile maximally. Each of these tasks is performed by the patient 80 three times to be sure a proper and usable sequence is recorded. In addition, each task begins with the patient's 80 face at rest, then moved to maximum movement and then returned to rest.
After the recording of the video images, the video cassette recorder 36 is conditioned in the playback mode. The video images are outputted by the video cassette recorder 36 over the signal path 38 and the video images are inputted into the frame grabber 40. The video images are passed through the frame grabber 40 and over the signal path 30 and inputted into the video monitor 28 where the video images are displayed.
The video images inputted into the frame grabber 40 are digitized and the frame grabber 40 outputs the digitized images or frames over the signal path 14. The digitized images are inputted into the disk storage 48.
The operator then conditions the processor 16 in the capture mode. In the capture mode, the operator can sequentially output the digitized images or frames from the disk storage 48 and input such digitized images or frames into the frame grabber 40. The frame grabber 40 converts each digitized image into an image in the analog format and the image in the analog format is outputted over the signal path 30 for display on the video monitor 28.
At this stage, the operator can select a select pair mode or a select sequence mode of operation by inputting to the processor 16 via the computer keyboard 22 a select pair command or a select sequence code.
Assuming the operator has selected the select pair mode and again referring to the facial movement example, the operator sequences through the frames. In this instance, the operator selects a control frame where the patient's 80 face is at rest and a moved frame where the patient's face has experienced maximum movement such as where the patient 80 has raised the patient's 80 eyebrow to maximum movement. The frames are selected by the operator inputting a capture code via the computer keyboard 22 to the processor 16. The selected frames are outputted by the frame grabber 40 over the signal paths 14 and 50 and stored in the disk storage.
After the control frame and the moved frame have been captured, the operator then inputs a process code to the processor 16 via the computer keyboard 22 to condition the processor 16 in the process mode. In the process mode, the processor 16 takes the control frame and the moved frame from the disk storage 48 and the processor 16 compares the control frame to the moved frame to produce a compared frame.
The processor 16 then produces a gray frame in a digital format. The gray image is produced from the compared frame by taking only the pixels having a light value above a predetermined light value, for example, above 22. The digitized gray frame is converted to a gray frame in an analog format by the frame grabber 40 and outputted over the signal path 30 and inputted into the display monitor 28 which displays the inputted gray frame. The gray frame provides somewhat of an outline of the patient's 80 head and facial features for the purpose of selecting areas of interest, in a manner which will be described in greater detail below.
After the gray frame has been displayed on the screen of the display monitor 28, the operator selects a center of the displayed image. In the example of analyzing facial motion, a center generally is picked about the end of a patient's 80 nose, a center 118 being shown in FIG. 3 for example. For the purpose of illustrating the selection of areas of interest, it is assumed that FIG. 3 represents the gray image displayed on the screen of the display monitor 28.
After the selection of center 118, a horizontal grid 120 and a vertical grid 122 are displayed on the screen of the display monitor 28 with the grids 120 and 122 intersecting at the selected center 118. Using the computer keyboard 22, the processor 16 is programmed to permit the operator to select an area of interest. Assuming the motion being analyzed is the raising of the patient's 80 eyebrow, the area of interest selected would be the area of interest 124 indicated by dashed lines in FIG. 3 which includes the area of the face which should have been moved or caused to be moved by the raising of the patient's eyebrow. In one embodiment, the operator selects an area of interest 126 on the right side of the patient's face. The process 16 automatically selects and outlines a corresponding area of interest 128 on the left side of the patient's 80 face. After the area of interest 124 has been selected, the operator then selects either the area of interest 126 or the area of interest 128 to be designated as the control position and the opposite area of interest 126 or 128 to be the moved position.
The operator then selects a threshold. The threshold is a value of the light intensity of a pixel and only pixels having a light value above the threshold are selected for analyzing. In the present embodiment of the invention, the operator has a number of thresholds which the operator can select depending on the desires of the operator and depending upon which threshold level gives the best comparison in the opinion of the operator. For example, a selected threshold might be 22.
After the threshold has been selected, the processor 16 then counts the number of pixels in the selected areas 126 and 128 having a light valve above the selected threshold, and determines the movement change using the counted pixels above the selected threshold in the area of interest 124. The processor 16 also determines a percent of movement between the left area of interest 126 and the right area of interest 128. For example, in one patient using one embodiment of the present invention with the selected areas being about the same as the selected areas 124, 126 and 128 and with the patient's 80 requested movement being the raising of the patient's 80 eyebrows, the following movement parameters were determined by the processor 16 and displayed on the screen of the computer monitor 18:
The numbers associated with the DIFF parameter in Table I above are arbitrary numbers representing the change in movement in the areas of interest 126 and 128 between the control frame and the changed frame. The TOP LEFT QUADRANT, TOP RIGHT QUADRANT, BOTTOM LEFT QUADRANT, and BOTTOM RIGHT QUADRANT listed in Table I are pixel counts that represent the number of pixel in the areas of interest 126 and 128 respectively having a light value above the selected threshold in the compare frame. The percent movement numbers in Table I above represent a ratio of the pixel counts between the left and the right side.
Assuming that the operator desires to select a sequence of frames to be analyzed, the process is identical to the process described before with respect to selecting pairs for analyzing, except instead of initiating a select pair code, the operator initiates via the computer keyboard 22 a select sequence code which is inputted into the processor 16 over the signal path 20. In the select sequence mode, the operator selects a sequence of frames to be analyzed. In one present embodiment of the invention, an operator can select up to a maximum of thirty-six frames for analyzing in the select sequence mode of operation.
For example and assuming that the requested movement again was the raising of the patient's 80 eyebrow, the operator might select a sequence of thirty-six frames starting with the patient's face in a rest position and then sequentially selecting a number of frames between the patient's face in the rest position through the maximum movement where the patient's eyebrows are raised and back to the rest position. Each of these frames is captured in the manner described before.
After the sequence of frames has been captured, the operator then selects a reference pair of frames from the selected frames, such as the first frame where the patient's face is at rest and one of the frames showing maximum movement of the patient's face where the eyebrow is in the raised position. The processor 16 compares the reference pair of frames and produces the gray frame in the manner described before with respect to the select pair mode of operation. After the gray frame has been produced, the gray frame is displayed on the computer monitor 18 and the areas of interest are selected in the manner described before with respect to the select pair mode of operation.
After the areas of interest have been selected, the operator then selects a control frame such as the first frame showing the patient's face in the control or rest position. The remaining frames in the sequence of frames are designated moved frames. The processor 16 then compares each moved frame with the control frame in the selected pairs sequentially. The reference frame and the moved frames are compared in sequence in a manner exactly like that described before with respect to the select pair mode of operation to produce a plurality of compare frames. After the processor 16 has compared sequentially the moved frames with the control frame in sequence, the processor 16 then outputs the comparison information similar to Table I, except the information would be included for the comparison of each of the moved frames with respect to the control frame in the selected sequence. For example, Table II below shows the output of the processor 16 where the requested movement was the raising of the patient's 80 eyebrow and the selected frames included the patient's 80 face at rest, through the movement of the eyebrows to a maximum moved position, and back to the patient's 80 face at rest.
Table II gives a quantitative picture of the movement of the patient's face from a rest position, to a raised eyebrow position and back to a rest position. Threshold and non-thresholded gray scale pixel intensity difference values are plotted against rest versus motion pairs. A pair is the reference control frame of the patient's face at rest compared to sequentially changed frames with the patient's face in a requested motion. Pair one, for example, would be the control frame subtracted from the next changed frame in sequence. Pair two, would be the control frame subtracted from the second changed frame in sequence. Pair three would be the control frame compared to the third changed frame in sequence.
By this plotting method, a dynamic function curve can be plotted, resulting from a given facial motion request, such as raising the brow, closing the eyes, wrinkling the nose, or smiling. This dynamic function has a steep contraction phase, a less steeply sloped end of contraction phase, a duration of maximum contraction, then a steeply sloping relaxation phase. This function has amplitude, duration, and an inscribed area as well.
A variety of data can be calculated from the dynamic curves, such as slope, intercept, duration, maximum amplitude, and integrated area under the curve. Each parameter of data then can be normalized to normal populations of varying ages to which similar data parameters from abnormal populations may be compared. By these methods, degrees of facial paresis and abnormal facial movements, such as synkinesis, may be quantitatively compared to normals and to the normal side of a given patient's face.
In some instances, one will analyze a left side and a right side of a patient in a single position or actually unchanged position. In this instance, the left side or one component part of the frame would be designated the reference position and the right side or one other component part of the frame would be designated the changed position, as those terms are used in the claims.
Changes may be made in the construction and the operation of the various components, elements and assemblies described herein and changes may be made in the step or the sequence of steps of the methods described herein without departing from the spirit and the scope of the invention as defined in the following claims.
Claims
What is claimed is:
1. A method for detecting facial motion of at least a portion of an individual's face comprising: producing a digitized frame of the individual's face in a reference position of the individual's face, identified as a digitized control frame; producing a digitized frame of the individual's face in a moved position of the individual's face, different from the position of the individual's face in the reference position, identified as a digitized changed frame, where the moved position is produced substantially by facial musculature movements of the individual; and comparing the digitized control frame with the digitized changed frame and determining shape changes over a defined surface area of the the individual's face from the reference position to the moved postion.
2. The method of claim 1 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format; grabbing a frame of the video images in the analog format showing the individual's face in the control position; and digitizing the frame of the video image in the analog format and in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in ana analog format; grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the video image in the analog format and in the changed position to produce the digitized control frame.
3. The method of claim 1 wherein the step of producing the digitized control frame further comprises: taking a image via a still frame camera of the individual's face in the control position to produce a still frame image in a digital format thereby producing the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: taking a image via a still frame camera of the individual's face in the changed position to produce a still frame image in a digital format thereby producing the digitized changed frame.
4. The method of claim 1 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual'face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; grabbing a frame of the video images int he analog format showing the individual's face in the control position; and digitizing the frame of the individual's face in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; and grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the individual's face in the changed position to produce the digitized changed frame.
5. The method of claim 1 wherein, prior to the step of comparing the digitized control frame with the digitized changed frame, the method further comprises: equalizing the digitized control frame to produce an equalized digitized control frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations; equalizing the digitized changed frame to produce an equalized digitized changed frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
6. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame further comprises: subtracting out the common areas of pixel values between the digitized control frame and the digitized changed frame to produce a digitized compare frame.
7. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: counting the pixels in the digitized compare frame having a brightness value above a predetermined threshold level to provide the indication of shape changes over the defined surface area of the individual's face from the control position to the changed position.
8. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; and counting the pixels having a brightness value above the predetermined threshold level in the area of interest.
9. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame, is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; designating a changed area of interest in the digitized compare frame; and counting the pixels having a brightness value above a predetermined threshold level in the control level in the control area of interest and counting the pixels having a brightness value above a predetermined threshold level in the changed area of interest to provide the indication of shape changes over the defined surface area in the control area of interest with respect to the changed area of interest.
10. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: enhancing all of the pixels in the digitized compare frame having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
11. The method of claim 1 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: replacing all of the pixels in the digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in the digitized compare frame having a brightness value above a predetermined threshold level whereby the pixels in the digitized compare frame having a brightness value above the predetermined threshold level appear more bright.
12. The method of claim 1 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: enhancing all of the pixels in each of the digitized compare frames having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
13. A method for detecting and quantifying facial motion of at least a portion of an individual's face comprising: producing a plurality of digitized frames of the individual's face as the individual's face is moved from one position to another position as a result substantially of facial musculature movements of the individual and designating one of the digitized frames as a digitized control frame and the remaining digitized frames as digitized changed frames; and comparing each digitized changed frame with the digitized control frame to determine the shape changes over a defined surface area for each comparison, with each of the determined shape changes over the defined surface area being indicative of shape changes over the defined surface area of the individual's face between the digitized changed frame compared with the digitized control frame.
14. The method of claim 13 wherein the step of producing the digitized frames further comprises: videoing using a video camera the individual's face as it is being moved from one position to another position to produce video images of the individual's face in an analog format; grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing each of the frames to produce the digitized frames with one of the digitized frames being designated the digitized control frame.
15. The method of claim 13 wherein the step of producing the digitized frames including the digitized control frame further comprises: taking a plurality of images via a still camera of the individual's face as the individual's face is being moved from one position to another position to produce a still frame image in a digital format thereby producing the digitized frames including the digitized control frame.
16. The method of claim 13 wherein the step of producing the digitized frames including the digitized control frame further comprises: videoing using the video camera the individual's face as the individual's face is moved from one position to another position to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputting via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; and grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing the frames of the individual's face to produce the digitized frames including the digitized control frame.
17. The method of claim 13 wherein, prior to the step of comparing the digitized control frame with the digitized changed frame, the method further comprises: equalizing the digitized control frame and the digitized changed frames to produce a digitized control frame and digitized changed frames with each having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
18. The method of claim 13 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: designating a control area of interest in each of the digitized compare frame; counting the pixels having a brightness value above a predetermined level in each of the areas of interest.
19. The method of claim 13 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: replacing all of the pixels in each digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in each of the digitized compare frames having a brightness value above a predetermined threshold level whereby the pixels in each of the digitized compare frames having a brightness value above the predetermined threshold level appear more bright.
20. The method of claim 13 wherein the step of comparing each digitized changed frame with the digitized control frame is further defined as producing a plurality of digitized compare frames with each digitized compare frame resulting from the comparison of one of the digitized changed frames with the digitized control frame, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: counting the pixels in each digitized compare frame having a brightness value of above a predetermined threshold level.
21. The method of claim 20 further comprising: plotting the number of pixels counted in each digitized compare frame in sequence as a function of time to provide the indication of shape changes over the defined surface area of the individual's face.
22. A method for detecting facial motion of an individual's face comprising: producing a digitized frame of the individual's face in a referenced position of the individual's face, identified as a digitized control frame; producing a digitized frame of the individual's face in a moved position of the individual's face identified as a digitized changed frame wherein the moved position is produced substantially by facial musculature movements of the individual; and comparing the digitized control frame with the digitized changed frame to determine shape changes over a defined surface area of the individual's face from the control position to the changed position.
23. The method of claim 22 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format; grabbing a frame of the video images in the analog format showing the individual's face in the control position; and digitizing the frame of the video image in the analog format and in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in ana analog format; grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the video image in the analog format and in the changed position to produce the digitized control frame.
24. The method of claim 22 wherein the step of producing the digitized control frame further comprises: taking a image via a still frame camera of the individual's face in the control position to produce a still frame image in a digital format thereby producing the digitized control frame; and wherein the step of producing the digitized changed image further comprises: taking a image via a still frame camera of the individual's face in the changed position to produce a still frame image in a digital format thereby producing the digitized changed frame.
25. The method of claim 22 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video media for outputting the video images in the analog format; grabbing a frame of the video images in the analog format showing the individual's face in the control position; and digitizing the frame of the individual's face in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video media for outputting the video images in the analog format; and grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the individual's face in the changed position to produce the digitized changed frame.
26. The method of claim 22 wherein, prior to the step of comparing the digitized control image with the digitized changed image, the method further comprises: equalizing the digitized control frame to produce an equalized digitized control frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations; equalizing the digitized changed frame to produce an equalized digitized changed frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
27. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame further comprises: subtracting out the common areas of pixel values between the digitized control image and the digitized changed frame to produce a digitized compare frame.
28. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: counting the pixels in the digitized compare frame having a brightness value above a predetermined threshold level to provide the indication of shape changes over the defined surface area of the individual's face from the control position to the changed position.
29. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; and counting the pixels having a brightness value above a predetermined threshold level in the area of interest.
30. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame, is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; designating a changed area of interest in the digitized compare frame; and counting the pixels having a brightness value above a predetermined threshold level in the control level in the control area of interest and counting the pixels having a brightness value above a predetermined threshold level in the changed area of interest to provide the indication of shape changes over the defined surface area in the control area of interest with respect to the changed area of interest.
31. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: enhancing all of the pixels in the digitized compare frame having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
32. The method of claim 22 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: replacing all of the pixels in the digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in the digitized compare frame having a brightness value above a predetermined threshold level whereby the pixels in the digitized compare frame having a brightness value above the predetermined threshold level appear more bright.
33. A method for detecting facial motion of an individual's face comprising: performing via the individual at least one of the following movements: (1) raising eyebrows from a rest position to a raised position, (2) smiling from a rest position to a smiled position, (3) closing the eyes from a rest position to a fully closed position of the eyes and (4) wrinkling the nose from a rest position to a fully wrinkled position; producing a digitized frame of the individual's face in a control position wherein the individual's face is at rest, identified as a digitized control frame; producing a digitized frame of the individual's face when the individual's face is positioned in at least one of the following moved positions: (1) raised eyebrows, (2) smiling, (3) eyes closed, and (4) wrinkled nose and identifying that digitized frame as a digitized changed frame where the moved positions are produced substantially by facial musculature movements of the individual; and comparing the digitized control frame with the digitized changed frame to determine shape changes over a defined surface area of the individual's face from the control position to the changed position.
34. The method of claim 33 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format; grabbing a frame of the video images in the analog format showing the individual's face in the control position; and digitizing the frame of the video image in the analog format and in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format; grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the video image in the analog format and in the changed position to produce the digitized control frame.
35. The method of claim 33 wherein the step of producing the digitized control image further comprises: taking a image via a still frame camera of the individual's face in the control position to produce a still frame image in a digital format thereby producing the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: taking a image via a still frame camera of the individual's face in the changed position to produce a still frame image in a digital format thereby producing the digitized changed frame.
36. The method of claim 33 wherein the step of producing the digitized control frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; grabbing a frame of the video images in the analog format showing the individual's face in the control position; and digitizing the frame of the individual's face in the control position to produce the digitized control frame; and wherein the step of producing the digitized changed frame further comprises: videoing using a video camera the individual's face to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputted via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; and grabbing a frame of the video images in the analog format showing the individual's face in the changed position; and digitizing the frame of the individual's face in the changed position to produce the digitized changed frame.
37. The method of claim 33 wherein, prior to the step of comparing the digitized control frame with the digitized changed frame, the method further comprises: equalizing the digitized control frame to produce an equalized digitized control frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations; equalizing the digitized changed frame to produce an equalized digitized changed frame having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
38. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame further comprises: subtracting out the common areas of pixel values between the digitized control frame and the digitized changed frame to produce a digitized compare frame.
39. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: counting the pixels in the digitized compare frame having a brightness value above a predetermined threshold level to provide the indication of shape changes over the defined surface area of the individual's face from the control position to the changed position.
40. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; and counting the pixels having a brightness value above a predetermined threshold level in the area of interest.
41. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame, is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: designating a control area of interest in the digitized compare frame; designating a changed area of interest in the digitized compare frame; and counting the pixels having a brightness value above a predetermined threshold level in the control level in the control area of interest and counting the pixels having a brightness value above a predetermined threshold level in the changed area of interest to provide the indication of shape changes over the defined surface area in the control area of interest with respect to the changed area of interest.
42. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises; enhancing all of the pixels in the digitized compare frame having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
43. The method of claim 33 wherein the step of comparing the digitized control frame with the digitized changed frame is further defined as producing a digitized compare frame, and wherein the step of comparing the digitized control frame and the digitized changed frame further comprises: replacing all of the pixels in the digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in the digitized compare frame having a brightness value above a predetermined threshold level whereby the pixels in the digitized compare frame having a brightness value above the predetermined threshold level appear more bright.
44. A method for detecting facial movements of an individual's face comprising: performing via the individual at least one of the following movements: (1) raising eyebrows from a rest position to a raised position, (2) smiling from a rest position to a smiled position, (3) closing the eyes from a rest position to a fully closed position of the eyes, and (4) wrinkling the nose from a rest position to a fully wrinkled position; producing a plurality of digitized frames of the individual's face as the individual's face is moved from one position to another position to moved positions and designating one of the digitized images as a digitized control frame and the remaining digitized frames as digitized changed frames where the moved positions are produced substantially by facial musculatures movements of the individual; and comparing each digitized changed frame with the digitized control frame to determine shape changes over a defined surface area for each comparison, with each of the determined shape changes over the defined surface area being indicative of shape changes over the defined surface area of the individual's face between digitized changed frame compared with the digitized control frame.
45. The method of claim 44 wherein the step of producing the digitized frames further comprises: videoing using a video camera the individual's face as it is being moved from one position to another position to produce video images of the individual's face in an analog format; grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing each of the frames to produce the digitized frames with one of the digitized frames being designated the digitized control frame.
46. The method of claim 44 wherein the step of producing the digitized frames including the digitized control frame further comprises: taking a plurality of images via a still camera of the individual's face as the individual's face is being moved from one position to another position to produce a still frame image in a digital format thereby producing the digitized frames including the digitized control frame.
47. The method of claim 44 wherein the step of producing the digitized frames including the digitized control frame further comprises: videoing using the video camera the individual's face as the individual's face is moved from one position to another position to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputting via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; and grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing the frames of the individual's face to produce the digitized frames including the digitized control frame.
48. The method of claim 44 wherein, prior to the step of comparing the digitized control frame with the digitized changed frame, the method further comprises: equalizing the digitized control frame and the digitized changed frames to produce a digitized control frame and digitized changed frames with each having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
49. The method of claim 44 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: designating a control area of interest in each of the digitized compare frame; counting the pixels having a brightness value above a predetermined level in each of the areas of interest.
50. The method of claim 44 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: enhancing all of the pixels in each of the digitized compare frames having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
51. The method of claim 44 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: replacing all of the pixels in each digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in each of the digitized compare frames having a brightness value above a predetermined threshold level whereby the pixels in each of the digitized compare frames having a brightness value above the predetermined threshold level appear more bright.
52. The method of claim 44 wherein the step of comparing each digitized changed frame with the digitized control frame is further defined as producing a plurality of digitized compare frames with each digitized compare frame resulting from the comparison of one of the digitized changed frames with the digitized control frame, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: counting the pixels in each digitized compare frame having a brightness value of above a predetermined threshold level.
53. The method of claim 52 further comprising: plotting the number of pixels counted in each digitized compare frame in sequence as a function of time to provide the indication of shape changes over the defined surface area of the individual's face.
54. A method for detecting facial motion of an individual's face comprising: producing a plurality of digitized frames of the individual's face as the individual's face is moved from one position to another position, a moved position, and designating one of the digitized images as a digitized control frame and the remaining digitized frames as digitized changed frames where the moved positions are produced substantially by facial musculature movements of the individual; and comparing each digitized changed frame with the digitized control frame to determine shape changes over a defined surface area for each comparison, with each of the determined shape changes over the defined surface area being indicative of shape changes over the defined surface area of the individual's face between the digitized changed frame compared with the digitized control frame.
55. The method of claim 54 wherein the step of producing the digitized frames further comprises: videoing using a video camera the individual's face as it is being moved from one position to another position to produce video images of the individual's face in an analog format; grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing each of the frames to produce the digitized frames with one of the digitized frames being designated the digitized control frame.
56. The method of claim 54 wherein the step of producing the digitized frames including the digitized control frame further comprises: taking a plurality of images via a still camera of the individual's face as the individual's face is being moved from one position to another position to produce a still frame image in a digital format thereby producing the digitized frames including the digitized control frame.
57. The method of claim 54 wherein the step of producing the digitized frames including the digitized control frame further comprises: videoing using the video camera the individual's face as the individual's face is moved from one position to another position to produce video images of the individual's face in an analog format and outputting the video images of the individual's face in the analog format; recording the video images in the analog format outputting via the video camera on a video medium in an analog format; playing back the video images recorded on the video medium for outputting the video images in the analog format; and grabbing a plurality of frames of the video images in the analog format showing the individual's face as the individual's face is being moved from one position to another position; and digitizing the frames of the individual's face to produce the digitized frames including the digitized control frame.
58. The method of claim 54 wherein, prior to the step of comparing the digitized control frame with the digitized changed frame, the method further comprises: equalizing the digitized control frame and the digitized changed frames to produce a digitized control frame and digitized changed frames with each having an average intensity throughout and with a predetermined distribution of pixel intensity variations.
59. The method of claim 54 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: designating a control area of interest in each of the digitized compare frame; counting the pixels having a brightness value above a predetermined level in each of the areas of interest.
60. The method of claim 54 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: enhancing all of the pixels in each of the digitized compare frames having a brightness level above a predetermined value whereby all of the pixels having a brightness value above a predetermined value appear to be more bright.
61. The method of claim 54 wherein the step of comparing the digitized control frame with each of the digitized changed frame further comprises producing a plurality of digitized compare frames with each digitized compare frame being produced as a result of the comparison of the digitized control frame with one of the digitized changed frames, and wherein the step of comparing each digitized changed frame with the digitized control frame further comprises: replacing all of the pixels in each digitized compare frame having a brightness value below a predetermined threshold level with a substantially null value and enhancing the pixels in each of the digitized compare frames having a brightness value above a predetermined threshold level whereby the pixels in each of the digitized compare frames having a brightness value above the predetermined threshold level appear more bright.
62. A method for detecting motion of an individual's face comprising: producing a digitized frame of of the individual's face and designating a part of the digitized frame as a digitized control frame and one other part of the frame as digitized changed frame; and comparing the digitized control frame with the digitized changed frame and determining shape changes over a defined surface area of the individual's face from the digitized control frame to the digitized changed frame where the shape changes between the digitized control frame and the digitized changed frame substantially are produced by facial muscular movements of the individual.
Patent Citations (9)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US4644582 | 1987-02-01 | Morishita et al. | |
| US4706296 | 1987-11-01 | Pedotti et al. | |
| US4805018 | 1989-02-01 | Nishimura et al. | |
| US4870692 | 1989-09-01 | Zuidaueld et al. | |
| US4899393 | 1990-02-01 | Morishita et al. | |
| US4951140 | 1990-08-01 | Ueno et al. | |
| US4989089 | 1991-01-01 | Chantelou et al. | |
| US4996594 | 1991-02-01 | Murayama | |
| US5008947 | 1991-04-01 | Yamada |
Non-Patent Literature (24)
- L. Jongkees, "Decompression of the Facial Nerve", Arch. Otolaryng. 85:473-78 (1967).
- N. May, "Facial Paralysis, Peripheral Type: Proposed Method of Reporting", Laryngoscope, 90:331-90 (1970).
- M. Karlan, "Contour Analysis in Plastic and Reconstructive Surgery Arch. Otolaryngol.", 105:670-80 (1979).
- L. Farkas, W. Bryson and J. Klotz, "Is Photogrammetry of the Face Reliable"?, Plastic Reconstructive Surgery 66:346-55 (1980).
- U. Fisch, "Surgery for Bell's Palsy", Arch. Otolaryngol, 107:1-11 (1981).
- B. Zide, B. Grayson and J. McCarthy, "Cephalometric Analysis: Part I," Plastic Reconstructive Surgery., 68:816-23 (1981).
- B. Zide, B. Grayson and J. McCarthy, "Cephalometric Analysis for Upper and Lower Midface Surgery: Part II," Plastic Reconstructive Surgery., 68:961-68 (1981).
- B. Zide, B. Grayson and J. McCarthy, "Cephalometric Analysis for Mandibular Surgery: Part III," Plastic Reconstructive Surgery. 69:155-64 (1982).
- M. Thorton & I. Pilowsky, "Facial Expressions Can Be Modelled Mathematically," Brit.J. Psychiat., 140:61-63 (1982).
- R. Kerbavaz, R. Hilsinger, and K. Adour, "The Facial Paralysis Prognostic Index," Otolaryngol. Head Neck Surgery. 91:284-89 1983.
- J. House, "Facial Nerve Grading Systems", Laryngoscope, 93:1056-69 (1983).
- P. Hilger, R. Webster, J. Hilger and R. Smith, "A Computerized Nasal Analysis System," Arch. Otolaryngol. 109:653-61. (1983).
- D. Brackmann and D. Barrs "Assessing Recovery of Facial Function Following Acoustic Neuroma Surgery," Otolaryngol. Head Neck Surg.
- J. House, "Facial Nerve Grading Systems", pp. 35-41. Proceedings of the Fifth International Symposim on the Facial Nerve.
- N. Yanagihara, "Grading System for Evaluation of Facial Palsy", pp. 41-42, Proceedings of the Fifth International Symposium on the Facial Nerve.
- M. Portmann, "Conclusions", Proceedings of the Fifth International Symposium on the Facial Nerve, pp. 43-44, Bordeaux, 1984, Masson Publishing U.S.A., Inc.
- J. House and D. Brackmann, "Facial Nerve Grading System", Otolary. Head Neck Surgery, 93:146-47 (1985).
- M. Ohyama, E. Obata and S. Furuta, "Color Electromyographic Topographic Analysis of Facial Movements" 185-90 (1985).
- S. Burres, "Facial Biomechanics: The Standards of Normal," Laryngoscope, 95:708-14 (1985).
- S. Burres, "Objective Grading of Facial Paralysis," Ann. Otol. Rhinol. Laryngol., 95:238-41 (1986).
- S. Burres, U. Fisch, "The Comparison of Facial Grading Systems" Arch. Otolaryngol. Head Neck Surgery, 112:755-58 (1986).
- M. Ohyama, E. Obata, S. Furuta, K. Sakamoto, Y. Ohbori and Y. Iwabuchi, "Face EMG Topographic Analysis of Mimetic Movements". (1988).
- B. Balough, M. Wessig, P. Waldhaust, W. Millesi, H. Millesi, W. Fibras, F. Fruhwald, "Photogrammetry of the Muscles of Facial Expression", Acta Anat. 133:183-87 (1988).
- M. Katsikitis, I. Pilowsky, "A Study of Facial Expression in Parkinson's Disease Using a Novel Microcomputer-Based Method". J. Neurol. Neurosurg. Psychiatry, 51: 362-66 (1988).