Combined digital and mechanical tracking of a person or object using a single video camera

[0001] Online broadcasting of lectures and presentations, live or on demand, is increasingly popular in universities and corporations as a way of overcoming temporal and spatial constraints on live attendance. For instance, at Stanford University, lectures from over 50 courses are made available online every quarter. University of California at Berkeley has developed online learning programs with “Internet classrooms” for a variety of courses. Columbia University provides various degrees and certificate programs through its e-learning systems. These types of on-line learning systems typically employ an automated lecture capturing system and a web interface for watching seminars online. FIG. 1 shows a screen shot of one such web interface 10. On the left hand side, there is a display sector 12 showing a video stream generated by the automated lecture capturing system being employed at the lecture site. Typically, this display is an edited video switching among a speaker view, an audience view, a local display screen view and an overview of the lecture room. Presentation slides of the lecture are displayed on the right in a slide sector 14 of the interface 10. The automated lecture capturing systems can vary greatly in their makeup. However, a typical example would include several analog cameras. For example, two cameras could be mounted in the back of the lecture room for tracking the speaker. A microphone array/camera combo could be placed on the podium for finding and capturing the audience. In some capture systems, each camera is considered a virtual cameraman (VC). These VCs send their videos to a central virtual director (VD), which controls an analog video mixer to select one of the streams as output.

[0002] Despite their success, these automated lecture capturing systems have limitations. For example, it is difficult to transport the system to another lecture room. In addition, analog cameras not only require a lot of wiring work, but also need multiple computers to digitize and process the captured videos. These limitations are partly due to the need for two cameras to track the speaker in many existing capture systems. One of these cameras is a static camera for tracking the lecturer's movement. It has a wide horizontal field of view (FOV) and can cover the whole frontal area of the lecture room. The other camera is a pan/tilt/zoom (PTZ) camera for capturing images of the lecturer. Tracking results generated from the first camera are used to guide the movement of the second camera so as to keep the speaker at the center of the output video. This dual camera system can work well, however it tends to increase the cost and the wiring/hardware complexity.

[0003] It is noted that while the foregoing limitations in existing automated lecture capturing systems can be resolved by a particular implementation of a combined tracking system and process according to the present invention, this system and process is in no way limited to implementations that just solve any or all of the noted disadvantages. Rather, the present system and process has a much wider application as will become evident from the descriptions to follow.

[0004] The present invention is directed toward a combined digital and mechanical tracking system and process for generating a video using a single digital video camera that tracks a person or object of interest moving in a scene. This is generally accomplished by operating the camera at a higher resolution than is needed for the application for which it is being employed, and cropping a sub-region out of the image captured that is output as the output video. The person or object being tracked is at least partially contained within the cropped sub-region. As the person or object moves within the field of view (FOV) of the camera, the location of the cropped sub-region is also moved so as to keep the subject of interest within its boundaries. When the subject of interest moves to the boundary of the FOV of the camera, the camera is mechanically panned to keep the person or object inside its FOV. As such tracking involves a combined digital and mechanical scheme.

[0005] One implementation of this combined digital and mechanical tracking technique involves, on a periodic basis, first detecting movement of the person or object being tracked in the last video frame captured by the video camera. It is then determined if the detected motion indicates the person or object is shown completely within a prescribed-sized portion the last frame captured. If it does, then a cropping region, which is the aforementioned prescribed-sized sub-region of the last frame that shows at least part of the person or object of interest, is established. This feature of finding the person or object being tracked within the last-captured frame of the video camera and establishing the cropping region is referred to as digitally tracking the person or object. However, if the detected motion indicates the person or object being tracked is not shown completely within the prescribed-sized portion the last frame captured, then the video camera is mechanically panned, with some possible exceptions, so as to show at least part of the subject of interest in a cropping region established in the last frame captured by the video camera after the mechanical panning is complete. The process of mechanically panning the camera to establish a cropping region containing the person or object of interest is referred to as mechanically tracking the person or object. Regardless of whether a digital or mechanical panning has occurred, the established cropping region is designated as the next frame of the video being generated. Thus, at each periodic time instance, another frame of the video is produced, showing the person or object of interest moving through the scene.

[0006] It should be noted that this Summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. In addition to the just described benefits, other advantages of the present invention will become apparent from the detailed description which follows hereinafter when taken in conjunction with the drawing figures which accompany it.

[0016] In the following description of embodiments of the present invention reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the invention may be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

1.0 The Computing Environment

[0017] Before providing a description of embodiments of the present invention, a brief, general description of a suitable computing environment in which portions of the invention may be implemented will be described. FIG. 1 illustrates an example of a suitable computing system environment 100. The computing system environment 100 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the invention. Neither should the computing environment 100 be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 100.

[0018] The invention is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known computing systems, environments, and/or configurations that may be suitable for use with the invention include, but are not limited to, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0019] The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

[0020] With reference to FIG. 1, an exemplary system for implementing the invention includes a general purpose computing device in the form of a computer 110. Components of computer 110 may include, but are not limited to, a processing unit 120, a system memory 130, and a system bus 121 that couples various system components including the system memory to the processing unit 120. The system bus 121 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus also known as Mezzanine bus.

[0021] Computer 110 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computer 110 and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computer 110. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0022] The system memory 130 includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM) 131 and random access memory (RAM) 132. A basic input/output system 133 (BIOS), containing the basic routines that help to transfer information between elements within computer 110, such as during start-up, is typically stored in ROM 131. RAM 132 typically contains data and/or program modules that are immediately accessible to and/or presently being operated on by processing unit 120. By way of example, and not limitation, FIG. 1 illustrates operating system 134, application programs 135, other program modules 136, and program data 137.

[0023] The computer 110 may also include other removable/non-removable, volatile/nonvolatile computer storage media. By way of example only, FIG. 1 illustrates a hard disk drive 141 that reads from or writes to non-removable, nonvolatile magnetic media, a magnetic disk drive 151 that reads from or writes to a removable, nonvolatile magnetic disk 152, and an optical disk drive 155 that reads from or writes to a removable, nonvolatile optical disk 156 such as a CD ROM or other optical media. Other removable/non-removable, volatile/nonvolatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 141 is typically connected to the system bus 121 through a non-removable memory interface such as interface 140, and magnetic disk drive 151 and optical disk drive 155 are typically connected to the system bus 121 by a removable memory interface, such as interface 150.

[0024] The drives and their associated computer storage media discussed above and illustrated in FIG. 1, provide storage of computer readable instructions, data structures, program modules and other data for the computer 110. In FIG. 1, for example, hard disk drive 141 is illustrated as storing operating system 144, application programs 145, other program modules 146, and program data 147. Note that these components can either be the same as or different from operating system 134, application programs 135, other program modules 136, and program data 137. Operating system 144, application programs 145, other program modules 146, and program data 147 are given different numbers here to illustrate that, at a minimum, they are different copies. A user may enter commands and information into the computer 110 through input devices such as a keyboard 162 and pointing device 161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 120 through a user input interface 160 that is coupled to the system bus 121, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor 191 or other type of display device is also connected to the system bus 121 via an interface, such as a video interface 190. In addition to the monitor, computers may also include other peripheral output devices such as speakers 197 and printer 196, which may be connected through an output peripheral interface 195. A camera 192 (such as a digital/electronic still or video camera, or film/photographic scanner) capable of capturing a sequence of images 193 can also be included as an input device to the personal computer 110. Further, while just one camera is depicted, multiple cameras could be included as input devices to the personal computer 110. The images 193 from the one or more cameras are input into the computer 110 via an appropriate camera interface 194. This interface 194 is connected to the system bus 121, thereby allowing the images to be routed to and stored in the RAM 132, or one of the other data storage devices associated with the computer 110. However, it is noted that image data can be input into the computer 110 from any of the aforementioned computer-readable media as well, without requiring the use of the camera 192.

[0025] The computer 110 may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 180. The remote computer 180 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 110, although only a memory storage device 181 has been illustrated in FIG. 1. The logical connections depicted in FIG. 1 include a local area network (LAN) 171 and a wide area network (WAN) 173, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0026] When used in a LAN networking environment, the computer 110 is connected to the LAN 171 through a network interface or adapter 170. When used in a WAN networking environment, the computer 110 typically includes a modem 172 or other means for establishing communications over the WAN 173, such as the Internet. The modem 172, which may be internal or external, may be connected to the system bus 121 via the user input interface 160, or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 110, or portions thereof, may be stored in the remote memory storage device. By way of example, and not limitation, FIG. 1 illustrates remote application programs 185 as residing on memory device 181. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0027] The exemplary operating environment having now been discussed, the remaining parts of this description section will be devoted to a description of the program modules embodying the invention.

2.0 The Combined Digital and Mechanical Tracking System and Process

[0028] The present combined digital and mechanical tracking system and process involves using a single digital video camera to track a person or object. This is accomplished by operating the camera at a higher resolution than is needed for the application for which it is being employed, and cropping a sub-region out of the image captured that is output as the output video. The person or object being tracked is at least partially shown within the cropped sub-region. As the person or object moves within the field of view (FOV) of the camera, the location of the cropped sub-region is also moved so as to keep the subject of interest within its boundaries. When the subject of interest moves to the boundary of the FOV of the camera, the camera is mechanically panned to keep the person or object inside its FOV. As such, the tracking involves a combined digital and mechanical scheme.

[0029] In the context of the previously-described limitations of existing automated lecture capturing systems, it can be seen that much of the cost and complexity of a dual, analog video camera tracking set-up is eliminated by the use of a single, digital PTZ video camera. For example, a network-type digital video camera can be employed, which takes advantage of existing Ethernet connections. In this way much of the wiring is eliminated and the system becomes much more portable. In addition, the digital nature of the camera eliminates any need for digitizing.

[0030] One implementation of this tracking technique is generally outlined in FIG. 3. In essence, this implementation of the tracking system and process involves, on a periodic basis, first detecting movement of the person or object being tracked in the last video frame captured by the video camera (process action 300). It is next determined if the detected motion indicates the person or object being tracked is shown completely within a prescribed-sized portion the last frame captured (process action 302). If it does, then a cropping region, which is a prescribed-sized sub-region of the last frame that shows at least part of the person or object of interest, is established (process action 304). This feature of finding the person or object being tracked within the last-captured frame of the video camera and establishing the cropping region is referred to as digitally tracking the person or object. However, if the detected motion indicates the person or object being tracked is not shown completely within the prescribed-sized portion the last frame captured, then the video camera is mechanically panned, with some possible exceptions, so as to show at least part of the person or object of interest in a cropping region established in the last frame captured by the video camera after the mechanical panning is complete (process action 306). The process of mechanically panning the camera to establish a cropping region containing the person or object of interest is referred to as mechanically tracking the person or object. In either case, the established cropping region is designated as the next frame of the video being generated (process action 308). Thus, at each periodic time instance, another frame of the video is produced, showing the person or object of interest moving through the scene.

[0031] The following sections will describe each module of the foregoing system and process in greater detail.

2.1 Motion Detection

[0032] As illustrated in FIG. 4, several regions are defined for use in the present tracking system and process. The first of these regions is the detection region 400. The detection region 400 represents a horizontal strip across the entire width of the FOV of the camera. In general, its lower and upper vertical boundaries are preset to encompass an area that it is believed any motion associated with the person or object of interest will occur. In the example image shown in FIG. 4, this detection region is set so as to contain a lecturer's upper body when in a standing position. As a lecturer will typically remain standing throughout the lecture, any motion associated with the lecturer would typically occur in the prescribed detection region.

[0033] If the scene containing the person or object being tracked also includes a secondary region of interest, the boundaries of this region are also preset. For example, in the context of the lecture example depicted in FIG. 4, the display screen at the front of the lecture room is of interest. As such, the horizontal and vertical boundaries of a “screen” region 402 are prescribed. The boundaries of the detection and screen regions will not typically change during the course of a tracking session. This is why their boundaries can be defined ahead of time. In tested embodiments, the heights of the lower and upper boundaries of each region 400, 402 were manually specified by a user, as were the locations of the lateral boundaries of the screen region 402. Notice that in the context of a lecture, this only needs to be done once for a given lecture room.

[0034] The remaining regions will move during the course of the tracking session and so are computed on a periodic basis as the session progresses. These regions include a cropping region 404, a safety region 406 and a motion region 408. The cropping region 404 defines the aforementioned sub-region of each frame of the captured video that is used to generate a frame of the output video. It is generally square or rectangular in shape and has an aspect ratio consistent with the desired format of the output video. For example, the captured video might have a resolution of 640×480, and the cropping region 404 might be a 320×240 sub-region of this view. In tested embodiments, the vertical position of the cropping region 404 is manually specified by a user and fixed. The user specifies a height that is anticipated will encompass the vertical excursions of the person or object being tracked within the vertical extent of the cropping region 404—at least most of the time. It is believed that in most applications that would employ the present system and process, using a fixed vertical height will be satisfactory while reducing the complexity of tracking a person or object of interest considerably.

[0035] The safety region 406 is a region contained within the cropping region 404 that is used to determine when a digital panning operation is to be performed as will be described shortly. This safety region 406 is defined as the region having lateral safety boundaries that are a prescribed distance W in from the lateral boundaries of the cropping region 404. The motion region 408 is an area computed based on motion detected in a frame. While the safety and motion regions 406, 408 are shown with top and bottom boundaries in FIG. 4 for ease in identification, these are not important to the present tracking system and process, and so need not be computed or prescribed by the user.

[0036] In regard to the motion region 408, it is noted that there have been many automatic detection and tracking techniques proposed that rely on detecting motion. While any of these techniques can be used, a motion histogram-based detection technique was adopted for use in tested embodiments of the present tracking system and process. This technique is simple, sensitive and robust to lighting variations. More particularly, consider a video frame captured at time instance t_n, n=0, 1, . . . . For each frame after the first, a frame difference is performed with the previous frame for those pixels in the prescribed detection region. All the corresponding pixel locations that exhibit an intensity difference above a prescribed threshold are then identified. In tested embodiments, the threshold was set to 15 (out of 256 grayscale levels), though such a threshold could vary for different rooms and their lighting conditions. The identified pixel locations in the current frame are designated as motion pixels. A horizontal motion pixel histogram is then generated. In essence this means using the count of the motion pixels found in each pixel column of the detection region to generate each respective bin of the histogram. The horizontal motion pixel histogram is then used to identify the horizontal segment of the current frame that contains the moving person or object of interest. More particularly, denote the histogram for the video frame captured at time instance t_n as h_k^tn, where k=1 . . . N and N is the number of bins which equals the number of pixel columns (e.g., 640 in a 640×480 frame). The person or object of interest, such as a speaker or lecturer, is deemed to be located in the “motion” segment Π_m^tn=(a_m^tn,b_m^tn) on the horizontal axis of the video frame captured at time instance t_n that satisfies the equation:

[0037] $\begin{array}{cc}\sum _{k\in {\Pi }_m^{t_n}}{h}_k^{t_n}=\sum _{k\in \varepsilon \left({\Pi }_m^{t_n},\delta \right)}{h}_k^{t_n}>\mathrm{.70}\sum _{k=1}^N{h}_k^{t_n},& \left(1\right)\end{array}$
where a_m^tn is the pixel column along the horizontal axis of the video frame captured at time instance t_n where the motion segment begins, b_m^tn is the pixel column along the horizontal axis of the video frame captured at time instance t_n where the motion segment ends, ε(Π_m^tn,δ) is an expansion operator which expands the motion segment Π_m^tn to both the left and right by δ. The above equation means that the motion segment is one that contains 70% of the motion pixels. In addition, it is one where no motion pixel will be added if the segment is expanded by δ. In tested embodiments, δ was set to 5 pixels, although a different value could be employed instead. If no segment fulfills both the above conditions, the motion segment is deemed to be the same as that computed for the previous time period, i.e., Π_m^tn=Π_m^tn-1. In the case where the motion detection procedure has just begun and there has been no segment that fulfills both the foregoing conditions as of yet, the motion segment is deemed to be “empty”. A graphical representation of an example horizontal motion pixel histogram 410 in the motion segment portion of the horizontal axis is shown superimposed on the image in FIG. 4.

[0038] It is noted that the tracking procedure does not begin until the motion detection region has reliably detected the location of the speaker. Once the speaker location is ascertained, an initial motion segment is produced. This initial motion segment is then used to start the tracking procedure.

2.2 Tracking

[0039] Given the motion detection results, a smooth output video that follows the person or object of interest can be generated using a combination of digital and mechanical tracking. Generally, with some exceptions, this is done by re-computing the location of the aforementioned cropping region at each time instance so as to keep the person or object of interest approximately centered in the region. As stated previously the cropping region becomes the output frame of the video being generated. To determine the new location of the cropping region at every time instance a tracking process is employed. More particularly, consider at time instance t_n, the detection procedure generates a motion segment Π_m^tn. This motion segmented Π_m^tn is used to compute the location of a cropping segment Π_c^tn=(a_c^tn, b_c^tn) where a_c^tn is the pixel column along the horizontal axis of the video frame captured at time instance t_n where the cropping segment begins and b_c^tn is the pixel column along the horizontal axis of the video frame captured at time instance t_n where the cropping segment ends. The vertical position of the cropping region is fixed and established prior to the tracking process as mentioned previously. Thus, the cropping segment completely defines the location of the cropping region.

[0040] The sections to follow will described how the cropping segment location is computed, first in the context of a digital tracking within the FOV of the video camera and then in the context of a mechanical tracking (e.g., mechanically panning the camera) if the person or object being tracked moves outside the FOV of the camera at its current position.

2.2.1 Digital Tracking

[0041] Rules collected from professional videographers suggest that a video camera following the movements of a person or object of interest should not move too often—i.e., only when the person or object moves outside a specified zone. This concept is adopted in the present tracking system and process. To this end, the aforementioned safety region is employed. More particularly, given the cropping segment computed at the last previous time instance (Π_c^tn-1), a safety segment is defined as Π_s^tn-1=(a_s^tn-1, b_s^tn-1), where a_s^tn-1−a_c^tn-1=b_c^tn-1−b_s^tn-1=W, and where W is the aforementioned prescribed distance in from the lateral boundaries of the cropping region and will be referred to as the safety gap. In tested embodiments, the safety gap was set to a constant value of W=40 pixels, although other values could be employed as well depending on what is being tracked and how fast it typically moves. The safety segment computed for the immediately preceding time instance is used to determine if a digital tracking operation will be performed at the current time instance or if the previous location of the cropping region is to be maintained. More particularly, if the motion segment computed for the current time instance is unknown or it falls completely inside this safety segment (i.e., the motion segment is empty (Π_m^tn=Ø) or is a subset of the safety segment (Π_m^tn⊂Π_s^tn-1), the location of the cropping region is left unchanged. Thus, the first rule of the present tracking system and process is:

• • [0042] Rule 1: If Π_m^tn⊂Π_s^tn-1, Π_c^tn-1=Π_c^tn-1.

[0043] However, if the motion segment computed for the current time instance is known (Π_m^tn-1≠Ø) and does not fall completely inside this safety segment computed for the previous time instance (Π_m^tn⊂/Π_X^tn-1), then two scenarios are considered. First, if Π_m^tn and Π_s^tn-1 do not overlap at all (Π_m^tn∩Π_s^tn-1=Ø), it is very likely that the person or object being tracked has completely moved outside the safety region. In this case, a digital panning operation to bring the subject back into the safety region is performed as will be described shortly. On the other hand, if Π_m^tn and Π_s^tn-1 partially overlap (Π_m^tn∩Π_s^tn-1≠Π_m^tn), this means the person or object being tracked is on one side of the cropping region but not out yet. In this latter case, a digital panning operation is not initiated unless this condition has persisted for more than a prescribed period of time T₀. In tested embodiments, T₀ was set to 3 seconds, although another period could be employed again depending on what is being tracked and how fast it is moving. By not immediately moving the cropping region when a person or object being tracked is straddling the safety segment boundary, the apparent motion of the camera in the output video is minimized in accordance with the aforementioned videographer rules.

[0044] Given the above, the second rule of the present tracking system and process can be characterized as:

• • [0045] Rule 2: If Π_m^tn∩Π_s^tn-1=Ø, or Π_m^tn∩Π_s^tn-1≠Π_m^tn for a period greater than T₀, digital panning is performed

[0046] Whenever a digital panning operation to bring the person or object being tracked back into the safety region is to be performed, it can be accomplished as follows. Without loss of generality, assume there is a need to digitally pan to the right (i.e., move the cropping region to the right within the current FOV of the video camera to bring the person or object being tracked back into the safety region). It is known that the right boundary of the motion segment is farther to the right than the right boundary of the safety segment—otherwise a digital panning operation would not have been initiated. Accordingly, it can be stated that b_m^tn>b_s^tn-1. Now, let d_right^tn=b_m^tn−b_s^tn-1. If the cropping region is moved to the right by d^tn at time instant t_n, the person or object being tracked will be found inside the safety region again. A similar procedure would be followed to digitally pan left, except in this case it is known that a_s^tn-1>a_m^tn and so d_left^tn=a_s^tn-1−a_m^tn.

[0047] Unfortunately, the foregoing scheme could make it appear that the camera view has “hopped”, instead of moving smoothly. Thus, while this method of digital tracking could be employed, a more elegant solution is possible. By observing professional videographers, it has been found that they can pan the camera very smoothly, even though the person or object being tracked may make a sudden motion. They do not pan the camera at a very fast speed, which implies that the panning speed should be limited. In addition, human operators cannot change their panning speed instantaneously. This could be mimicked by employing a constant acceleration. To this end, movement of the cropping region during a digital panning operation could alternately be computed by applying a unique constant acceleration, limited speed (CALS) model. More particularly, let the moving speed of the cropping region at time instance t_n be v^tn(v^tn≧0). The moving speed can be computed as:
v^tn=min(v^tn-1+αs^tn(t_n−t_n-1), v_max).  (2)
where s^tn is the sign of d^tn, α is a prescribed constant acceleration (e.g., 150 pixels per square second) and v_max is a prescribed maximum panning speed (e.g., 80 pixels per second).

[0048] Given the moving speed at the time instance t_n, the cropping segment at t_n can be computed as:
Π_c^tn=S(Π_c^tn-1,v^tn(t_n−t_n-1)),  (3)
where S(Π,x) is a shift operator that shifts the last previously computed cropping segment Π_c^tn-1 horizontally by the shifting distance x to the right or left depending on if d_right^tn or d_left^tn was used.

[0049] The computed cropping segment location is then used along with the prescribed vertical height of the cropping region to determine the location of the cropping region within the overall captured frame associated with the current time instance t_n.

[0050] It is noted that in the case of the first time instance at the beginning of the tracking procedure, the aforementioned initial motion segment is used to define a cropping segment location that acts as the “previous” cropping segment location for the above computations. In one embodiment, the location of this initial cropping segment is established as the prescribed width of the segment centered laterally on the center of the initial motion segment.

2.2.2 Mechanical Tracking

[0051] The digital tracking procedure described above can track the person or object being tracked inside the FOV of the camera. However, the person or object of interest may move out of the FOV of the camera at its current position. In such cases, the video camera needs to be mechanically panned to follow the person or object. Notice that before the person or object being tracked moves out of the FOV of the camera, the motion detection procedure should report a motion segment located around the boundary of a captured video frame. Given this, the decision to initiate a mechanical tracking operation can be made very simple. Generally, if any part of the current motion segment comes within a prescribed distance of the boundary of the current captured video frame on either side, a mechanical panning operation may be initiated.

[0052] During the mechanical panning operation, the motion detection procedure described previously cannot detect the person or object being tracked with any reliability. Therefore, the last computed location of the cropping region remains fixed until the mechanical panning has stopped. The amount of mechanical panning relies on the camera zoom level. In essence, the goal is to pan the camera in the direction of the person or object being tracked just enough so as to center the person or object within the temporarily fixed location of the cropping region. For example, assume the width of the person or object being tracked at the current zoom setting of the video camera is approximately 120 pixels. Thus, before the mechanical panning begins, the center of the speaker is about 60 pixels inward from one of the boundaries of the capture frame under consideration. In addition, assuming the cropping region is 320 pixels wide and the captured frame is 640 pixels wide, the width of the cropping region extend either from 0 to 320 or from 320 to 640. With these parameters, if the camera is mechanically panned 100 pixels in a direction that will bring the center of the next captured frame closer to the person or object being tracked, that person or object will be approximately in the middle of the cropping region, assuming the location of the cropping region is not changed in relation to the overall frame from its location in the last previous time instance and the person or object being tracked remains static. Thus, each mechanical panning operation initiated at the aforementioned zoom level would entail panning the camera in the appropriate direction by 100 pixels. The panning distance can be readily calculated for other zoom levels either on the fly or ahead of time. A quick way to make the panning distance calculation is to subtract the width of the person or object being tracked at the current zoom level (w_z) from the width of the cropping region (w_c) and then dividing by two (i.e., (w_c−w_z)/2).

[0053] It is also noted that continuous mechanical panning can be distracting to the viewer. As such, in one embodiment of the present tracking system and process, two sequential mechanical panning motions have to be separated by a prescribed time interval. For example, in tested embodiments, the time interval was set to 3 seconds, although a shorter or longer time period could be employed. When a mechanical panning is called for, but precluded due to the prescribed time interval test, at each time instance prior to reaching the prescribed time interval, a frame of the video being generated is created using the cropping region location associated with the last previous time instance.

[0054] In view of the foregoing, the third rule of the present tracking system and process associated with mechanical panning could be characterized as:

• • [0055] Rule 3: Mechanical panning of the video camera is initiated if Π_m^tn∩[(A,ε)∪(B−ε,B)]≠Ø and no previous mechanical panning operation has been perform in the time period T_mp,
    where ε is a small value corresponding to the aforementioned prescribed distance to the boundary of the captured video frame, A refers to the boundary of the frame on the left side, B refers to the boundary of the frame on the right side and T_mp is the aforementioned prescribed minimum time interval between mechanical panning operations. It is noted that in tested embodiments of the present tracking system and process, ε was measured in pixel columns and set to 2 columns. Thus, in this example, if the edge of the motion segment comes within 2 pixel columns of the captured frame boundary on either side at time instance t_n, a mechanical panning operation maybe initiated.
    2.2.3 The Combined Digital and Mechanical Tracking Process Flow

[0056] The following is a description of one embodiment of a process flow for performing the combined digital and mechanical panning operation described above. Referring to FIGS. 5A-E, the process begins by defining the detection region based on user input (process action 500). In addition, a secondary region of interest can be optionally defined at this point, again based on user input (optional process action 502). The purpose for this designation will be described in the next section. The vertical height of the cropping region is established as specified by the user (process action 504). Once all the preliminary matters are complete, the tracking process proceeds by determining if the first time instance has been reached (process action 506). If not, the action 506 is repeated. When it is determined the first time instance has been reached, the location of the motion segment is computed (process action 508). As indicated earlier, in one embodiment of the present tracking process, computing the motion segment for the current time instance involves computing the segment using a motion histogram-based detection technique. If a segment is found, it is designated as the motion segment for the current time instance. However, if no motion segment can be found, then either the last previously computed motion segment is designated as the current motion segment, or if no previous segment exists, the motion segment is designated as being “empty”. Once the motion segment has been established, it is determined if any part of the current motion segment comes within a prescribed distance of the boundary of the current captured video frame on either side (process action 510). If not, in process action 512, the location of the safety segment is computed based on the cropping region for the last previous time instance (or in the case of the first time instance based on the initial motion segment location). The location of the cropping region for the current time instance is then computed. More particularly, it is first determined if the current motion segment is empty (process action 514). If it is not, then it is determined if the motion segment is completely within the extent of the last computed safety segment (process action 516). If the motion segment is contained within the safety segment, or if it was determined that the motion segment is empty, then in process action 518 the location of the current cropping segment is set equal to the location of the last previously computed cropping segment (or in the case of the first time instance based on the initial motion segment location). However, if in process action 516 it is determined that the motion segment is not completely within the extent of the last computed safety segment, then it is determined if the motion segment is completely outside of the last computed safety segment or if it is partially overlapping the extent of the safety segment (process action 520). In the case where it is overlapping, it is determined if the period of time that the overlap condition has existed exceeds the prescribed period T₀ (process action 522). If not, then the location of a current cropping segment is set to the location of the last previously computed cropping segment or in the case of the first time instance, to the location of a cropping segment based on the initial motion segment location (process action 524). If, however, it is determined in process action 520 that the motion segment is completely outside of the last computed safety segment, or if it is determined in process action 522 that the period of time that the overlap condition has existed does exceed the prescribed period T₀, the side (i.e., right or left) of the last computed safety segment that the current motion segment is adjacent to or straddling, is identified (process action 526). Next, the distance between the corresponding side of the motion segment (i.e., right or left) and the identified side of the safety segment is computed (process action 528). It is noted that in one embodiment of the present tracking process, the current cropping segment location can be computed as the last previous location of the cropping segment shifted in the direction (i.e., right or left) of the identified side of the safety segment by the distance computed in process action 528. Alternately, the previously-described CALS technique can be employed to produce a smoother result. The process flow outlined in FIG. 5D will reflect this later procedure, although it is not intended that the tracking process be limited to this alternative. In the CALS technique, the next process action 530 is to compute the moving speed of the cropping region at the current time instance. As indicated previously, the moving speed will be the lesser of the prescribed maximum velocity, or the velocity computed for the last time instance, increased by the product of the prescribed acceleration and the difference in time between the current and last time instances and given the sign (i.e., + or −) of the distance computed in process action 528. The current cropping segment location is then computed as the last previous location of the cropping segment shifted in the direction (i.e., right or left) of the identified side of the safety segment by a shifting distance (process action 532). As described previously, the shifting distance is computed as the moving speed of the cropping region at the current time instance multiplied by the difference in time between the current and last time instances. No matter how the current cropping segment is established (see process actions 518, 524 and 532), the next process action 534 is to establish the location of the current cropping region using the just computed cropping segment and the prescribed vertical height of the region. However, if in process action 510 it was determined that some part of the current motion segment falls within the prescribed distance of one of the side boundaries of the last captured video frame, it is determined if a mechanical panning operation has been performed within the prescribed minimum time interval (process action 536). If it has, then no mechanical panning is performed and at each time instance prior to the expiration of a prescribed time interval, a frame of the video being generated is created using the cropping region location associated with the last previous time instance (process action 538). However, if no mechanical panning operation has occurred within the prescribed minimum time interval, then the mechanical panning distance is computed for the current camera zoom level (process action 540). This is followed in process action 542 by mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured closer to the person of object being tracked. The location of the current cropping region is then established as that computed for the last previous time instance (process action 544). It is then determined if the next time instance has been reached (process action 546). If not, process action 546 is repeated. Once the next time instance is reached, it is determined if the video session is ongoing (process action 548). If so process actions 508 through 548 are repeated, as appropriate. Otherwise the process ends.

2.3 Intelligent Pan/Zoom Selection

[0057] Mixing digital and mechanical tracking by applying Rules 1-3 together can provide very satisfactory results. However, there are additional aesthetic aspects that can be included in the present tracking system and process that go beyond just following the person or object of interest. Namely, the aforementioned secondary area of interest can be handled differently and the camera zoom level can be automated. Both of these features would further enhance the viewability of the video produced.

2.3.1 Secondary Area of Interest

[0058] As indicated previously, there may be an area in a scene being videotaped that is of interest to the viewer aside from the person or object being tracked. In some cases, it is desired to present this area in a special way when it is shown in the output video. For example, professional videographers suggest that if a speaker walks in front of a presentation screen, or if there are animations displayed on the screen, the camera should be pointed toward that screen. Traditionally this is handled using a dedicated video camera that captures images of just the screen. The output of this separated camera is employed in the video produced at the appropriate times. A similar scheme is followed for any secondary area of interest. However, it is possible to mimic the function of this separate, dedicated camera using the same camera that tracks the person or object of interest as described above.

[0059] To accomplish the foregoing task, the previously described tracking system and process needs to be modified somewhat. More particularly, the area of interest should be kept inside the FOV of the camera as much as possible, without eliminating the person or object being tracked from the view. This allows the secondary area of interest to be cropped from the overall frame and used as desired in the video being produced. To fulfill the above requirement, it will sometimes be necessary to mechanically pan the video camera toward the secondary area of interest to keep it in view, even though the previously described tracking procedure may dictate that a digital panning operation be performed to track the person or object of interest. This is because a digital panning operation would not bring more of the secondary area of interest into the overall captured frame, whereas a mechanical panning operation toward that area would result in more of it being captured. In view of this, the modified procedure entails giving priority to performing a mechanical tracking operation whenever the following three conditions are satisfied. First, the secondary area of interest is not fully inside the FOV of the camera. Second, there is a need to perform digital panning towards where the secondary area of interest is due to motion of the person or object being tracked. And third, performing a mechanical tracking operation as described previously will not result in the person or object being tracked being eliminated from the FOV of the camera at its new position. In such scenarios, the digital panning operation is overridden in favor of a mechanical panning operation.

[0060] In view of the foregoing, an optional fourth rule of the present tracking system and process could be characterized as:

• • [0061] Rule 4: A mechanical panning of the camera is commenced if,
    • [0062] a) Π_sa^tn∩(A,B)≠Π_sa^tn where Π_sa^tn is the location of a horizontal segment at time instance t_n corresponding to the lateral extent of the secondary area of interest;
    • [0063] b) A digital panning towards the secondary area of interest is needed in accordance with the aforementioned Rule 2; and
    • [0064] c) Π_m^tn⊂(η,B) if panning to the right, or Π_m^tn⊂(A,B−η) if panning to the left, where η is a prescribed number of pixels in the horizontal direction. In tested embodiments η=160, but could be another value. For example, η could be based on the zoom level of the camera.

[0065] The following process flow description adds an embodiment of the foregoing secondary area of interest feature to the overall tracking process outlined in FIG. 5A-E. Only those process action associated with this feature will be discussed. Referring to FIG. 6, the process begins after process action 534 of FIG. 5D is completed, and entails determining if the secondary area segment is completely contained within the video frame captured at time instance t_n (process action 600). If it is, then the process of FIG. 5E continues starting with process action 546. If, however all or part of the secondary area segment is found to be outside the video frame captured at time instance t_n, then it is determined if the just computed current cropping segment location is closer to the secondary area segment than the cropping segment location computed for the last previous time instance (process action 602). If not, then the process of FIG. 5E continues starting with process action 546. However, if it is closer, then it is next determined if the current motion segment will likely remain within the FOV of the frame of the video captured after a mechanical pan operation is performed (process action 604). If it will, then the process of FIG. 5D continues starting with process action 536 eventually resulting in a mechanical panning operation. If not, then the process of FIG. 5E continues starting with process action 546 and no mechanical panning operation is commenced.

2.4 Automatic Zoom Level Control

[0066] A person or object being tracked will behave differently depending on the circumstances. For example, one lecturer will often behave very differently from another lecturer when giving a lecture. Some lecturers stand in front of their laptops and hardly move; others actively move around, pointing to the slides, writing on a whiteboard, switching their slides in front of their laptop, etc. For the former type of lecturers, it is desirable to zoom in more, so that viewer can clearly see the lecturer's gestures and expressions. In contrast, for the latter type of lecturers, it is not desirable to zoom in too much because that will require the video camera to pan around too much during the tracking operation. With this in mind, it is possible to include an optional automatic zoom level control feature in the present tracking system and process that will handle the different types of movement likely to be encountered when tracking a person or object. This feature is based on the level of activity associated with the person or object being tracked. However, unlike the tracking portion of the present system and process, it would be distracting to a viewer if the zoom level of the camera could be changed at every time instance. It is better to only do it once in a while.

[0067] More particularly, let the period between zoom adjustments be zoom period T₁. The total distance that the person or object being tracked moved over a period T₁ is computed. One way of accomplishing this task is to sum the number of pixels in the horizontal direction that the cropping region moved over the zoom period T₁. Recall at time instance t_n, the movement is v^tn(t_n−t_n-1) for digital panning. In view of this let:
u=Σ_tnεT1v^tn(t_n−t_n-1)+M×u₀,
where M is the number of mechanical pannings in period T₁ and u₀ is the number of pixels moved during each mechanical panning. Note that u₀ will depend on the zoom level used during period T₁ and is determined as described previously.

[0068] At the end of each time period T₁, the zoom level of the video camera is adjusted by the following rule:

• • [0069] Rule 5: At the end of each time period T₁, change the zoom level according to:

[0070] $z_{\mathrm{new}}=\{\begin{array}{cc}\max \left(z_{\mathrm{old}}-\Delta z,z_{\min }\right)& \mathrm{if}u>U_1\\ \min \left(z_{\mathrm{old}}+\Delta z,z_{\max }\right)& \mathrm{if}u<U_2\\ z_{\mathrm{old}}& \mathrm{otherwise}\end{array}$
Here z_new is the new zoom level and z_old is the old zoom level. Δz is the step size of zoom level change. z_max and z_min are maximum and minimum zoom levels. U₁>U₂ are activity thresholds. In tested embodiments, the time period T₁ was set to 2 minutes. The Δz, z_max, z_min, U₁ and U₂ values are set based on the camera involved and the amount of motion anticipated. As a default, the smallest zoom level z_min can be used as the initial zoom setting. It was found that the zoom level would stabilize within 5-10 minutes in a lecture environment. It is noted that the foregoing parameter values were tailored to a lecture environment. In other environments, these values would be modified to match the anticipated movement characteristics of the person or object being tracked.

[0071] Given the foregoing, one embodiment of the automatic zoom level control feature according to the present system and process can be implemented as described in following process flow. Referring to FIG. 7, the process starts by setting the video camera to a prescribed initial zoom level at the beginning of a tracking session (process action 700). It is next determined if a zoom period has expired (process action 702). If not, no action is taken. However, when the period expires, the total horizontal distance that the cropping region moved during the last zoom period is computed (process action 704). The zoom level that is to be used for the next zoom period is then computed based on the total distance computed for the last zoom period (process action 706). The video camera is then zoomed to the computed zoom level if it is different from the level used in the last zoom period (process action 708). Next, it is determined if the tracking session is still ongoing (process action 710). If so, process actions 702 through 710 are repeated. Otherwise, the process ends.

[0072] In addition to automatically controlling the zoom level periodically based on the movement of the person or object being tracked, the automatic zoom level control feature can include a provision for re-acquiring the trackee should the motion detection procedure fail. Referring to FIG. 8, it is determined if the motion detection procedure reports no motion for a prescribed motion period (process action 800). In tested embodiments, the motion period was set to 5 seconds. However, depending on the nature of the anticipated movement of the person or object being tracked, another period may be more appropriate. If it is determined that no motion has been detected for the prescribed period, then the video camera is zoomed out to its aforementioned minimum level (process action 802) and the process ends. By zooming the camera out, the motion detection procedure then has a better chance of re-detecting the person or object being tracked because of the larger field of view.

[0073] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.