Apparatus and methods for resolution of entry/exit conflicts for security monitoring systems

[0002] 1. Field of the Invention

[0003] The present invention relates generally to security monitoring systems and, more particularly, to a security monitoring system having a computer vision system for recognizing individuals entering and leaving a structure to resolve entry/exit conflicts.

[0004] 2. Prior Art

[0005] Security monitoring systems of the prior art, particularly residential security systems, utilize a box that monitors contact sensors for doors and windows and one or more infra-red sensors for area monitoring. When a contact is triggered or an infra-red sensor triggers, an alarm is sounded and a signal is sent via a data link such as a phone line to a central monitoring site. The central monitoring site typically initiates a set of phone calls, to the homeowner, to work, and/or to a designated neighbor to determine if the alarm signal was due to an unauthorized intruder or just to an accidental triggering by a family member or other authorized occupant of the structure.

[0006] If the alarm signal cannot be resolved by the phone calls, it is passed to the local police department. 95% of the calls passed to the police department turn out to be “false alarms” in the sense that they were not due to an unauthorized intruder.

[0007] Significant portions of “false alarms” are caused by what is referred to as entry/exit conflicts. For instance, in the situation of a residential alarm system, the homeowner or other authorized occupant of a residence often arms the security system while leaving the residence and shortly thereafter realizes that they have forgotten something in the residence. As they return to the residence, they enter without disarming the system thereby causing an alarm to be sounded and/or an alarm signal to be sent to a central monitoring site. Similarly, the homeowner may arm the security system and remain inside the residence, such as during the night and may thereafter leave to get something outside the residence, i.e., the morning paper, thereby triggering a false alarm.

[0008] In view of the prior art, there is a need for a security monitoring system, which resolves these and other types of entry/exit conflicts.

[0009] Therefore it is an object of the present invention to provide a security monitoring system, which resolves the entry/exit conflicts of the prior art security monitoring systems.

[0010] It is a further object of the present invention to provide a security monitoring system that monitors the entrance and exit of individuals to a structure, records key image data, and matches an exiting individual against an entering individual.

[0011] Accordingly, a security monitoring system is provided. The security monitoring system comprises; an exit and entrance camera located at the exit and entrance, respectively of a structure; a detector for detecting the exit or entry of an individual into or from the structure; an image recording system for storing images from the exit and entrance cameras; and a computer vision system for analyzing the stored images transmitted from the image recording system using a predetermined criteria to determine if the exiting and entering individuals are the same. Preferably, the exit and entrance cameras are either video cameras or the images are video images or digital cameras and the images are still images.

[0012] The security monitoring system preferably further comprises means for transmitting an alarm signal and/or at least a portion of the stored images to a remote site if the exiting and entering individual is determined not to be the same. Alternatively, the security monitoring system can transmit an alarm signal and/or at least a portion of the stored images if the exiting and entering individual is the same.

[0013] In a preferred implementation of the security monitoring system of the present invention, the predetermined criteria used to determine if the exiting and entering individuals are the same comprises matching descriptive features of the entering and exiting individual such as a color, shape, or texture distribution of the stored images.

[0014] In yet a more preferred implementation of the security monitoring system of the present invention, the security monitoring system further comprises: a database for storing image data for each authorized individual of the structure; and a recognition system for comparing the stored images from the entry and entrance camera with the stored image data in the database and for determining if the exiting and entering individual is one of the authorized individuals. Preferably, the stored image data in the database are face images and the recognition system is a face recognition system. In the more preferred implementation, the security monitoring system further comprises means for transmitting an alarm signal and/or at least a portion of the stored images to a remote site if the exiting and entering individual is determined not to be the same and the exiting and entering individual is determined not to be one of the authorized individuals.

[0015] Also provided is a method for performing security monitoring of a structure. The method comprising the steps of; capturing images at both an exit and entrance of the structure; before or after the capturing step, detecting the exit or entry of an individual into or from the structure; storing the images from the exit and entrance camera; and analyzing the stored images using a predetermined criteria to determine if the exiting and entering individuals are the same.

[0016] Preferably, the method further comprises the step of transmitting an alarm signal and/or at least a portion of the stored images to a remote site if the exiting and entering individual is determined not to be the same. Alternatively, the security monitoring system can transmit an alarm signal and/or at least a portion of the stored images if the exiting and entering individual is the same.

[0017] In a preferred implementation of the methods of the present invention, the method further comprises the steps of: storing image data for each authorized individual of the structure; and comparing the stored images from the entry and entrance with the stored image data for determining if the exiting and entering individual is one of the authorized individuals. The preferred implementation of the methods of the present invention further comprises the step of transmitting an alarm signal and/or at least a portion of the stored images to a remote site if the exiting and entering individual is determined not to be the same and the exiting and entering individual is determined not to be one of the authorized individuals.

[0018] These and other features, aspects, and advantages of the apparatus and methods of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings where:

[0019] FIG. 1 illustrates a schematical view of the security monitoring system of the present invention.

[0020] FIG. 2 illustrates a flow chart of a first entry/exit conflict to be resolved by the security monitoring system of FIG. 1.

[0021] FIG. 3 illustrates a flow chart of a second entry/exit conflict to be resolved by the security monitoring system of FIG. 1.

[0022] Although this invention is applicable to numerous and various types of security monitoring systems, it has been found particularly useful in the environment of residential security monitoring systems. Therefore, without limiting the applicability of the invention to residential security monitoring systems, the invention will be described in such environment.

[0023] Referring now to FIG. 1, there is illustrated a security monitoring system of the present invention, generally referred to by reference numeral 100. The security monitoring system 100 includes an exit and entrance camera 102, 104 located at the exit 102aand entrance 104aof a door 106, respectively, of a residential home or a commercial building, generally referred to herein as a structure 108. The door 106 is shown as a typical exterior door by way of example only and not to limit the scope of the invention. Those of skill in the art will recognize that any thing providing an entry or exit into or from a structure can be the subject of the present invention, such as a sliding door, a roll-up door, or a garage door. The exit and entrance cameras 102, 104 are preferably video cameras which supply a sequence of video frames, but can alternatively be digital cameras supplying a sequence of still pictures. Furthermore, the exit and entrance cameras 102, 104 can be static or pan-tilt-zoom (PTZ) types. Both the sequence of video frame and still pictures are generally referred to herein as images or image data. The exit and entrance cameras 102, 104 are shown mounted near the door 106 and facing towards the exit and entrance 102a,104a,respectively. However, this configuration is illustrated for its simplicity. Preferably, the exit and entrance cameras 102, 104 are mounted in the door 106 as is described in co-pending U.S. patent application Ser. No. 09/734,780 which is incorporated herein by its reference.

[0024] The security monitoring system also includes an alarm system 110. The alarm system 110 has a detector for detecting the exit or entry of an unknown individual through the door 106 and into or from the structure 108. Although the unknown individual may turn out to be an authorized individual, at the time of exit or entry he or she has yet to be classified as either authorized or unauthorized. The detector can be any device known in the art for detecting the operation of a door 106, such as a magnetic door contact sensor 112 positioned on the door 106 or an infra-red sensor 114 located near the door 106 of the structure. Alternatively, the detector can be a sensor located on the door-knob of the door 106 or a background change detection system such as is also disclosed in co-pending U.S. patent application Ser. No. 09/734,780. The detector can also detect the act of arming the alarm system, or turning the door-knob or key tumbler on the door 106 or any device which can serve as a detector.

[0025] An image recording system 116 stores the images of the unknown individual from the exit and entrance cameras upon the triggering of the detector. That is, if the detector is triggered such that an alarm signal would ordinarily be transmitted to a central monitoring site 118, the image recording system 116 stores the images from the exit and entrance cameras 102, 104. Alternatively, the image recording system 116 can store all of the images in a buffer where new image data overwrites old image data such that a predetermined amount of current image data is always stored and available. It is preferred that the image recording system 116 is a computer or any other processor having a storage device such as a hard drive and an image capture card. However, those skilled in the art will recognize that the image recording system 116 can be of any type known in the art without departing from the scope and spirit of the present invention.

[0026] A computer vision system 120 analyzes the stored images of the entering and leaving individuals transmitted from the image recording system 116 using a predetermined criteria to determine if the exiting and entering individuals are the same. Preferably, the computer vision algorithm extracts descriptive features of the entering and leaving individuals. These descriptive features are preferably based on one or a combination of the color, texture, or shape distribution of the images. Preferably, the descriptive features are focused on a particular area of the entering and leaving individuals, such as his or her clothes. The descriptive features of the images of the individuals entering and leaving are extracted and compared. If there is no match between the entering and leaving individuals, an alarm signal is triggered and transmitted to the central monitoring site 118 via a data link 119, such as a telephone line (POTS). Preferably the image data of the entering or leaving individual is also transmitted to the central monitoring site 118. It is preferred that computer vision system 120 is a system such as that disclosed in co-pending U.S. application Ser. No. 09/734,821, which is incorporated herein by its reference. However, those skilled in the art will recognize that the computer vision system 120 can be of any type known in the art without departing from the scope and spirit of the present invention, such as that disclosed in Raja et al., “Segmentation and Tracking Using Colour Mixture Models”, 3^rdAsian Conference on Computer Vision, Vol. 1, pp. 607-614, Hong Kong, China, January 1998, which is also incorporated herein by its reference.

[0027] In a preferred implementation of the security monitoring system 100 of the present invention, there is also included a database 122 for storing image data, such as the face, for each authorized individual of the structure. In the case of a residential home, the faces of the people living in the home are recorded, preferably by the exit and entry cameras 102, 104 and stored in the database 122. Where the computer vision system 120 determines that the exiting and entering individuals are not the same, instead of triggering an alarm, the security monitoring system 100 can alternatively utilize a recognition system 124 which compares the stored images of the entering and leaving individuals from the entry and entrance cameras with the stored image data (e.g., faces) of the authorized individuals in the database 122. If there is no match between the stored images of the entering and leaving individuals from the entry and entrance cameras with the stored image data of the authorized individuals in the database 122 then the system proceeds as described above by triggering an alarm signal and/or transmitting image data of the entering or leaving individual to the central monitoring site 118. Face recognition algorithms and systems are well known in the art, such as the one described by S. Gutta et al., Face Recognition,Sixth International Conference on Computer Vision, pgs. 646-651, IEEE, Jan. 4-7, 1998, Mumbai, India.

[0028] If either the comparison of the entering and leaving individuals from the computer vision system 120 or the comparison of the stored images of the entering and leaving individuals from the entry and entrance camera with the stored image data of the authorized individuals in the database 122 results in a match, the alarm signal is not triggered (i.e., an exit or entry that would ordinarily trigger an alarm is ignored). Alternatively, the alarm signal can be transmitted upon the “unauthorized” exit or entry and an ignore signal can follow if it is determined that the entering and exiting individuals are the same. Furthermore, even if the exiting and entering individuals are determined to be the same, the alarm signal and/or a portion of the image data can be transmitted to the central monitoring site. For instance, if the unknown individual is determined to be the maid, who is authorized to be in the structure, the image data can later support or deny a subsequent allegation that the maid stole property from the structure.

[0029] A combination of face as well as descriptive features of the unknown individual can also be used as a way to verify the unknown individual as being authorized. This can be done either sequentially as discussed below or in parallel.

[0030] The security monitoring system 100 has been described with reference to FIG. 1 in general terms without reference to a particular entry/exit conflict. Two particular exit/entry conflicts will now be described with reference to FIGS. 2 and 3. However, those skilled in the art will realize that the general description above is intended to cover any such exit/entry conflict and is not therefore limited to any one particular entry/exit conflict or to the entry/exit conflicts described with regard to FIGS. 2 and 3.

[0031] Referring now to the flowchart of FIG. 2, there is illustrated a method for providing the security monitoring system of the present invention where the alarm system 110 is “armed” and an entrance into the structure 108 will cause the triggering of the alarm and transmission of an alarm signal to the central monitoring site 118. Such is the case where an authorized person “arms” the alarm system 110, exits the structure 108, and returns a short time later without disarming the alarm system 110 because they have forgotten something inside the structure 108.

[0032] At step 200, the exit and entrance cameras 102, 104 capture images from the exit and entrance 102a,104aof the door 106 of the structure 108. Alternatively, the cameras can come on to capture image data upon the occurrence of an event, such as the turning of a door knob or key tumbler on the door 106 or triggering a sensor on or near the door 106 or detecting through background models. At step 202 one or more of the detectors 112, 114 check for the detection of an exiting individual. If there is a detection of an exit (shown schematically as path 204b), the exit camera 102 will store a last predetermined portion of the video frames or still images from a buffer (which may also be database 122) at step 204. Alternatively, the exit camera 102 can capture images and the image recording system 116 store those images upon the detection of an event, such as when the authorized person “arms” the alarm system in which case the exit camera 102 can capture and record the arming of the alarm system. If there is no detection of an exit, the flow chart proceeds along path 204aand the cameras 102, 104 continue to capture images or alternatively return to a dormant state before capturing images upon the occurrence of the event.

[0033] The security monitoring system 100 then looks for a re-entry into the structure at step 208. If the alarm system 110 detects a re-entry into the structure within a predetermined time limit (shown schematically as path 208b), the images from the entry camera 104 are stored or taken from the buffer into the image recording device 116 at step 210. If there is no re-entry, the system continues along path 208ato step 200 to capture images from the exit and entrance cameras.

[0034] The stored images from the exit and entry cameras 102, 104 are then analyzed by the computer vision system 120 at step 212 to determine if the descriptive features therein match at step 214. If there is a match (i.e., it is determined that the same person who left, later re-entered) the system continues along path 214aand the detection of the re-entry is ignored at step 218. If there is not a match, the system can follow along either path 214cor 214b.If the system follows along path 214c(shown as a dotted line) to step 216, an alarm signal is transmitted preferably along with the stored images from the entrance camera to the central monitoring site 118. However, in case the exiting person re-enters with a different descriptive feature, as is the case where a color distribution is used and the person exits with a jacket and re-enters without it, a further face recognition can be performed at step 220. At step 220, the stored image data, such as the faces of all of the people authorized to have access to the structure, are compared to the stored images from the entrance camera 104. At step 222, it is determined if there is a match. If there is no match, the system proceeds along path 222bto step 216 where an alarm is triggered. If there is a match, the system proceeds along path 222ato step 218 where the detection of the re-entry is ignored or alternatively, along path 222c(shown as a dotted line) where the alarm signal and/or image data can also be transmitted to the central monitoring site as discussed above.

[0035] Referring now to the flowchart of FIG. 3, there is illustrated a method for providing the security monitoring system of the present invention where the alarm system 110 is “armed” and an exit from the structure 108 will cause the triggering of the alarm and transmission of an alarm signal to the central monitoring site 118. Such is the case where an authorized person “arms” the alarm system during the night, exits the structure upon waking without disarming the alarm system, and returns a short time later, perhaps with the morning paper. Similar reference numerals in FIG. 3 denote similar steps as described above with reference to FIG. 2.

[0036] At step 200, the exit and entry cameras 102, 104 capture images from the exit and entrance 102a,104aof the door 106 of the structure 108. At step 202 one or more of the detectors 112, 114 check for the detection of an exiting individual. If there is a detection of an exit (shown schematically as path 204b), the exit camera 102 stores the video frames or still images at step 204. If there is no detection of an exit, the flow chart proceeds along path 204aand continues to capture images or the cameras 102, 104 return to a dormant state until the occurrence of an event.

[0037] The security monitoring system 100 then looks for a re-entry into the structure at step 208. If the alarm system 110 detects a re-entry into the structure within a predetermined time limit (shown schematically as path 208b), the last portion of images from the entry camera 104 are stored from a buffer into the image recording device 116 at step 210. If there is no re-entry, the system continues along path 208cto step 216 where an alarm signal is triggered.

[0038] The stored images from the exit and entry cameras 102, 104 are then analyzed by the computer vision system 120 at step 212 to determine if the descriptive features therein match at step 214. If there is a match (i.e., it is determined that the same person who left, later re-entered) the system continues along path 214aand the detection of the exit is ignored at step 300. If there is not a match, the system can follow along either path 214cor 214b.If the system follows along path 214c(shown as a dotted line) to step 216, an alarm signal is transmitted preferably along with the stored images from the exit camera to the central monitoring site 118. However, in case the exiting person re-enters with a different descriptive feature, as is the case where a color distribution is used and the person exits with a jacket and re-enters without it, a further face recognition can be performed at step 220. At step 220, the stored image data, such as the faces of all of the people authorized to have access to the structure, are compared to the stored images from the entrance camera 104. At step 222, it is determined if there is a match. If there is no match the system proceeds along path 222bto step 216 where an alarm is triggered. If there is a match, the system proceeds along path 222ato step 300 where the detection of the exit is ignored or alternatively, along path 222c(shown as a dotted line) where the alarm signal and/or image data can also be transmitted to the central monitoring site as discussed above.

[0039] Those skilled in the art will recognize that the system illustrated in FIGS. 2 and 3 resolve the entry/exit conflicts described above. That is, a false alarm will not be triggered where an authorized person leaves and arms the alarm system but re-enters without disarming the alarm system or leaves while the alarm system is armed and returns.

[0040] While there has been shown and described what is considered to be preferred embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.