Elevator Passenger Detector and Door Control System Which Masks Portions of a Hall Image to Determine Motion and Court Passengers

The present invention is related to elevator passenger detection.

The level of performance of an elevator system is a function of a number of parameters. Among these parameters is the extent to which passengers and prospective passengers of the elevator system are happy with the service they receive from the elevators. This, in turn, is a function of a number of parameters, many of them relating to the amount of time that the passenger spends waiting to be delivered to his destination. This time spent waiting is, in turn, a function of a number of time-based parameters. One of these time-based parameters is the waiting time, that is, the amount of time from when a hall call button is pushed to when the elevator arrives. Another time-based parameter is the ride time between when the elevator car leaves the floor or a hall call button is pushed to the time when the car arrives at the destination floor of the passenger. Between a) the time that the elevator car arrives at the floor to receive the passenger who has pressed a hall call button and b) the time when the elevator car leaves is c) time spent opening elevator doors, holding the doors open so that the passenger can enter and closing the doors. The time spent holding the elevator doors open so that the passenger can enter is called the door dwell time. The door dwell time is typically a fixed quantity which is large to accommodate the great length of time required for a slow-moving passenger or a group of passengers to enter the car. Where a fast-moving passenger quickly enters the car from the hall, that passenger as well as all the passengers already in the car is forced to wait for the elevator door dwell time associated with a slow moving passenger to expire. Thus, this fast-moving passenger has an unnecessarily long waiting time because the door dwell time is fixed to a value suitable to some other slow moving passenger.

In addition, the amount of time that the passenger spends waiting to be delivered to his destination is a function of the efficiency of the elevator dispatching system which is in turn a function of the amount of passenger traffic known to the dispatching system.

Objects of the present invention include elevator passenger detection, door control in response to passenger detection, and a count of the number of elevator passengers.

According to the present invention, by motion detection, using an image signal of an area in front of an elevator door, and masking out of the image signal increasingly with time greater portions of the image signal when no motion is detected. In further accord with the present invention, elevator door control is varied in response to elevator passenger detection. In further accord with the present invention, the elevator passengers are counted.

A first advantage is that the elevator door dwell time is variable. A second advantage is that the average waiting time for passengers in an elevator system decreases because the car is not parked at a floor with doors open to receive passengers that are merely standing in front of the elevator doors without making movements toward the elevator. A further advantage is that the doors are not kept open for a fixed time associated with the time for a slow-moving passenger to enter the elevator. A fourth advantage is that passengers moving near the elevator may be distinguished from those who are not.

A feature of the invention is an image-based motion detection system capable of distinguishing between approaching passengers and closing doors yields the advantage that, in response to approaching passengers, closing elevator doors are commanded to reverse so that an approaching passenger can enter the elevator. Moreover, this feature leads to the further advantage that reversal of the elevator doors is not falsely triggered in response to door movement rather than passenger movement.

FIG. 1 is a partial block diagram illustrating the hardware used in the present invention and its positional relationship to an elevator door. The unmasked area is shaded.

FIG. 2 is a flow chart for control in response to detection of motion.

FIG. 3 is a top view of an elevator car showing boundaries of an elevator door sill mask.

FIG. 4 is a top view of an elevator car showing boundaries of an elevator door sill mask where the boundaries are changed so that the unmasked area is narrowed as compared with the sill mask of FIG. 3 because the elevator doors have moved towards a closed door position.

FIG. 5 is a top view of an elevator car showing boundaries of a hall mask. The unmasked area is shaded.

FIG. 6 is an overhead view of an elevator car wherein the hall mask changed so that the unmasked area is decreased in size relative to that illustrated in FIG. 5 to exclude from the video signal a person to cause that person to not be detected.

FIG. 7 is an overhead view of an elevator car showing boundaries of a hall mask. The unmasked area is shaded.

FIG. 8 is an overhead view of an elevator car wherein the hall mask changed so that the unmasked area is decreased relative to the hall mask of FIG. 7 because the position of the elevator doors is different from the position of the elevator doors in FIG. 7.

FIG. 9 is a functional block diagram of the hardware and software for implementing the present invention detection called for by the routine of FIG. 2.

FIG. 10 is a flow chart for creation of the hall mask.

FIG. 11 is a flow chart for creation of the sill mask.

FIG. 12 shows the process of summing the hall and sill masks with the difference image (including a sill difference image and a hall difference image).

FIG. 13 shows a video image for an empty hall.

FIG. 14 shows the video image for the hall of FIG. 13 with three people in it.

FIG. 15 shows the masked difference image.

FIG. 16 shows the masked difference image after threshold comparison.

FIG. 17 shows the masked difference image and the number of peaks representative of the number of moving objects in the unmasked viewing area.

FIG. 1 shows a video camera mounted to a door controller which is itself mounted on an elevator car. The video camera is directed so that it receives a picture from the hall between two elevator doors. Only a right door, car, and hall, are shown. An image processing people detector receives a video signal from the video camera and a door position signal from an encoder mounted to a motor shaft of an door controller. This allows for detecting a person in the viewing area and providing a control signal to the door controller for commanding closing and opening of the elevator doors in response to detection of a person. The image processing people detector includes an image processing board exemplified by the one available from Androx of Canton, Mass.--the Androx ICS 400 image processing board with Androx ADSP-2100 software.

In addition, the number of people viewed may be counted.

FIG. 2 is a flow chart for door control. It is the main software routine. The method represented by the flow chart is triggered when the elevator car stops at a landing. Timing of the door control in response to detected motion within the viewing area depends on a condition: whether the doors are fully open because they were reversed to the fully open position. Control of the doors in response to detected motion is delayed a first delay time if the doors have been commanded fully open in normal operation and not in response to a reversal. If no reversal has occurred, the doors are commanded to stay open until the first delay time has expired. In addition, a door dwell timer is set to 2 seconds and the view depth/breadth is set to a maximum. The depth/breadth of view is altered by changing the boundaries of a hall mask for excluding portions of the image provided by the camera from image processing for the purposes of motion detection. This is explained with regard to FIGS. 5-12. The result is that motion detection is limited to images provided within unmasked areas.

If the doors are fully open as a result of a reversal, or following the above steps, then if motion is detected, the depth of the area viewed by the camera is reduced. This sequence is repeated until no motion is detected or the door dwell time expires.

If no motion is detected in the unmasked hall or sill areas, the doors are commanded to close as apparently there is nobody moving near the elevator.

If the door dwell time expires and motion is detected within the sill region, then the doors are commanded to stay open as apparently someone is moving between the doors.

Alternate to varying the view depth, the view breadth may be altered.

If the doors are fully closed, the reversal flag is cleared. If the doors have been commanded to close and are not fully closed and motion is detected, the doors are commanded open and the reversal flag set. If the doors are not fully closed and the door dwell time has not expired, the view depth/breadth is reduced. Detection of motion in the unmasked area causes the doors to open and the reversal flag to be set. If the doors are not fully closed and no motion is detected clearing the reversal flag awaits the door dwell fully closed condition. The main routine is not again executed until the doors again become fully open.

Motion detection includes reception of a video image by the camera (FIG. 1) and masking out portions of the received image with the hall and sill masks so that only motion in a portion of the viewing area near the elevator is considered for varying door dwell time and arriving at a people count within the unmasked area.

FIGS. 3, 4, 5, 6, 7, 8 illustrate the hall and sill masks. Two masks are required: a hall mask and a sill mask. Generation of the hall mask includes generating five points for forming a pentagon. Generation of the sill mask includes generating four points for forming a rectangle. Both may be done using a graphics software package like the one available from Androx associated with the Androx image processing board.

FIG. 3 is a top view of an elevator car showing boundaries of the sill mask. The doors are shown fully open. Four points, sill-near-right corner (SNRC), sill-near-left corner (SNLC), sill-far-right corner (SFRC) and sill-far-left corner (SFLC) are shown. The location of these points is used, along with door position provided by the encoder, in a routine (described below in FIG. 11) to generate the sill mask. The unmasked area where moving objects may be detected is shaded. The area covered by the sill mask is not limited to the actual physical sill including the car side and hall side of the sill; it may extend out into the hall. Selection of the area of the sill mask is restricted only by the points SNRC, SFLC, SFRC, and SNLC and the door position. This leads to the opportunity to change the sill mask by changing SNRC, SFLC, SFRC, and SNLC or changing the door position.

FIG. 4 is a top view of an elevator car showing boundaries of the sill mask which has narrowed the unmasked area (shaded area) as compared with FIG. 3 because the elevator doors have moved towards a more closed position.

FIG. 5 is a top view of an elevator car showing boundaries of a hall mask.

FIG. 6 is an overhead view of an elevator car wherein the hall mask is increased in size relative to that illustrated in FIG. 5 to exclude from the video signal a person to cause that person to not be detected. In other words, the depth of view of the unmasked area is smaller than in FIG. 5. This illustrates the steps in FIG. 2 demonstrating that if motion is detected, the depth of view is reduced. The purpose of doing this is to distinguish between passengers moving towards the elevator and passengers that are not moving towards the elevator. Alternate to varying the view depth, the view breadth may be altered.

FIG. 7 is an overhead view of an elevator car showing boundaries of the hall mask. Here, the doors are in a more closed position than in FIG. 5. The area of the unmasked area is reduced accordingly and the reduction has occurred near the door.

FIG. 8 is an overhead view of an elevator car wherein the hall mask is increased relative to the hall mask of FIG. 7 and accordingly the unmasked area is still smaller as the doors are nearly fully closed.

The mask boundary parameters for both the sill and hall masks are below. Four feet is an exemplary door sill width.

FIG. 9 is a block diagram for varying door dwell time in response to detected motion in the sill area and/or the hall and for counting moving objects in the sill area or hall. Those portions of the invention implemented in hardware (HW) are distinguished from those portions of the invention implemented through software (SW) by the dotted line. The camera is, for example, a charge coupled device (CCD) which receives a picture of the hall and sill for providing a video image to a frame grabber located on the Androx image processing board in the image processing people detector of FIG. 1. Each time that a video image is provided to the frame grabber, a new image from the video camera is digitized and stored in the frame grabber.

The frame grabber provides a new frame of the new image from the video camera to a frame comparator and to an image buffer for storage. The image buffer provides an old frame, that is, an image previously provided by the frame grabber, to the frame comparator. The old image provided on initialization of the invention is of a sill and hall clear of people and objects except the elevator and hall system.

The frame comparator compares the new frame with the old frame for providing a difference frame. The comparison of the new frame with the old frame provides an indication of motion in the area seen by the video camera. The value of the difference frame signal indicates motion. The frame comparison is performed within a CPU of the image processing board and is a pixel-to-pixel subtraction and threshold comparison of the new image to the old image. This threshold comparison in the frame comparator is the first of two threshold comparisons in the motion detection process.

Where the video camera provides a black and white video image, the result of frame comparison is a difference image with a given pixel set to either white or black.

Hard limiting through a threshold comparison may be used before or after the difference frame is provided. Where hard limiting is used before the difference image is provided, the function implemented is of EXCLUSIVE ORing of the new image and the old image for providing a difference image having white or black pixels. Alternatively, the new image may be subtracted from the old image and then hard limited through the first threshold comparison for providing a difference image of only white and black pixels. The latter is shown in FIGS. 13-17.

The resulting difference image has the appearance of a silhouette, that is only the moving objects appear in the image; non-moving objects are black pixels.

The difference image is provided to a summer. In response to a door position signal--provided by the door encoder for purposes of locating the door edges--a mask generator provides two masks including a sill mask and a hall mask. From the location of the camera in the elevator car and the location of the left and right corners of the landing door edges, the location of the landing floor visible to the camera is computed in response to door movement.

A hall mask is generated for excluding from consideration motion outside a given part of the viewing area and for cancelling detection of the door movement. Cancelling door movement prevents a false reading of a moving passenger caused when door motion is mistaken for passenger motion. The breadth of the hall mask is adjusted in response to change in location of the door edges. The depth of view of the hall mask area is also changed in response to detection of motion in the unmasked area of the hall. If the elevator doors begin to close, the unmasked area narrows by reducing each of the unmasked areas: the unmasked area in the hall and the unmasked area between the doors over the door sill.

The mask generator stores the mask boundary parameters. The mask boundary parameters are provided to a sill mask generator and a hall mask generator which provide the sill mask and a hall mask to a summer. The sum of the hall and sill masks is then added to the difference image.

The size of the masks may be changed if a motion detected signal is provided to the mask generator. If the doors have not yet fully closed or opened, the mask boundary parameters may be varied. Varying the x coordinates of HFLC or HFRC varies the breadth of the unmasked hall area. Varying the y coordinates of HFLC and HFRC varies the depth of the unmasked hall area. Varying the y coordinates of SFLC, SFRC, SNRC and SNLC varies the height of the unmasked sill area. The width of the unmasked sill area varies with door position.

If no motion is detected, the depth of view is reduced. This reduction is repeated periodically with each run of the main routine of FIG. 2 if no motion is detected in each of successively smaller unmasked areas until the doors are closed.

The difference image and hall and sill masks are logically summed. All pixels that are masked out are made black.

The result of this is that only so much of the silhouettes that appear in the unmasked areas are available.

After masking out a portion of the difference image using the hall mask and the sill mask, a masked difference image indicative of motion in the unmasked area of the hall is provided to a cluster analyzer. There, adjacent lit pixels are combined into clusters. Pixels are lit if they are bright or white rather than dark or black. Small clusters are discarded because they may represent noise rather than true brightness changes indicative of motion in the unmasked area. This cleans up the video image. This is done by adjacent lit pixels being counted and the count compared to a second threshold. If the count in any cluster exceeds the second threshold, motion is assumed to have occurred in the unmasked area. If the lit pixel count in any cluster is less than the second threshold, then no motion is judged to have occurred in the unmasked area of the hall.

A motion detected signal is provided (FIG. 2) if the second threshold is exceeded. The motion detected signal is provided to a depth of view block for indicating that motion was or was not detected as indicated by the threshold count having been exceeded or having not been exceeded. In response, the mask generator varies the magnitude of the view depth. As the amount of time during which no motion is detected increases, the view depth decreases. Motion detection is more fully described below. Alternate to varying the view depth, the view breadth may be altered.

In addition, the motion detected signal is provided to a peak detector for peak detection. The number of peaks detected is equal to the number of moving objects in the unmasked area, that is generally a people count.

FIG. 10 is a flow chart for generation of the hall mask. (Shown in FIGS. 5-8). The software package used is Androx software to be used in conjunction with the Androx image processing board. The inputs are the location of the video camera, the door edge locations as provided by the encoder, and the field of view locations stored in memory on the image processing board and illustrated in the table "Mask Boundary Parameters". Step 1 designates the camera location C(x,y). Step 2 computes the slope S1, from the camera to the corner of the right door edge (RD), given by the encoder, of a line 1. Step 3 computes the intersection I1(x,y) of line 1 with the line X=RC. Steps 4 and 5 are analogous to steps 2 and 3, but for the left side. Steps 6 and 7 together determine whether the unmasked area will be pentagonal (FIGS. 5-7) or triangular (FIG. 8). In Step 6, if the Y component of intersection I1 is less than the Y component of RC(x,y) then the intersection I1 is on the right side of the unmasked area and therefore point 2 of the unmasked area is intersection I1 and point 2 is RC. Then, the unmasked area is pentagonal.

Else, an intersection I3 of the line 1 is computed as identically, both points 2 and 3. Therefore the unmasked area is triangular. Step 7 is analogous to step 6 but for the left side.

FIG. 11 is a flow chart for creation of a sill mask. Inputs to the routine are illustrated in the table. These points are shown in FIGS. 3,4. The mask has four points.

In step 1, point 1 is designated as determined by the x coordinate of RD and the Y coordinate of SNRC. In Step 2, point 2 is designated as RD. Step 3 designates point 3 as LD. Step 4 designates as point 4 the x coordinate of LD and the y coordinate of SNR.

FIG. 12 shows the process of summing the sill and hall masks with the difference image (including a sill difference image and a hall difference image).

FIG. 13 shows a video image for an empty hall. This image is the old one.

FIG. 14 shows the video image for the hall of FIG. 13 with three people in it, all standing within the unmasked area. This is the new image.

FIG. 15 shows the difference frame.

FIG. 16 shows the difference frame after the first threshold comparison in the frame comparator.

FIG. 17 shows the number of peaks representative of the moving number of objects in the unmasked viewing area is three. The number of peaks is generally a people count.

Various changes may be made herein without departing from the invention. Alternate to the image processing board from Androx and separate camera is the "Imputer" available from VLSI VISION LTD of Edinburgh, Scotland which includes a camera and image processing board in the same package. In addition, alternative to the use of a video signal from a video camera is the use of an infrared signal or ultraviolet signal; masking of either type signal is included in the present invention.