Video surveillance system based on high resolution network cameras capable of concurrent transmission of multiple image formats at video rates

[0001] This application claims the benefit of U.S. Provisional Application Nos. 60/524,490 filed Nov. 21, 2003, 60/524,449 filed Nov. 21, 2003, 60/524,448 filed Nov. 21, 2003, and 60/524,488 filed Nov. 21, 2003, the contents of which are incorporated by reference herein.

[0002] Also incorporated by reference herein are the following applications which were filed on Jul. 14, 2004: Ser. Nos. 10/890,870, 10/890,911, 10/890,912, 10/890,913, 10/890,914 and 10/890,915.

[0003] Also incorporated by reference herein are the following applications filed on even date herewith:

[0004]

	Priority
Title	Application
HIGH RESOLUTION NETWORK VIDEO CAMERA	60/524,449
WITH MASSIVELY PARALLEL IMPLEMENTATION
OF IMAGE PROCESSING, COMPRESSION AND
NETWORK SERVER
METHOD FOR ACCURATE REAL-TIME	60/524,488
COMPENSATION FOR CHANGING ILLUMINATION
SPECTRA IN DIGITAL VIDEO CAMERAS

[0005] Most currently installed video surveillance systems are based on NTSC/PAL/SECAM analog video cameras with camera resolution rigidly fixed by the corresponding video standard. Thus, to inspect fine detail of the scene, NTSC surveillance systems have to rely on expensive optical zoom. However, higher optical zoom inevitably corresponds to reduced field of view of the camera, requiring the operator to make a choice between higher level of detail and higher coverage of the area under surveillance.

[0006] On the other hand, leveraging high-speed LAN infrastructures and employing high resolution (multi-megapixel) network cameras in video surveillance allows to avoid many of the shortcomings associated with low resolution NTSC cameras. Network video cameras rely on packet-oriented digital image transmission are not limited to any particular image resolution and frame rate. This opens the door for the development of high-resolution video surveillance systems that are overall superior to existing NTSC systems and provide video-rate multi-format functionality previously unavailable in video surveillance.

[0007] Present invention discloses a number of modes of operation of video surveillance system based on previously disclosed by the author network cameras with massively parallel image processing, compression and network server.

[0008] The subject of the present invention is video surveillance system that comprises a number of high resolution network video cameras, wherein all or some of said cameras are capable of concurrent transmission of multiple image formats at video rates, common to these cameras video server/computer and system-level software, wherein said server is configured to acquire the images from said cameras, to store said images in digital archives, to display said images on one or more monitors and to respond to the requests of local and remote users by sending to said users requested images over local or wide area networks.

[0009] The other subject of the present invention is video system comprising one or more high resolution network video cameras, computer and monitor, wherein said computer is configured to receive images from said video cameras by means of computer network and wherein said computer is configured to display said images on said computer monitor and wherein said display of said images comprises display mode wherein full field of view image and a sub-window of said image are displayed on said monitor simultaneously.

[0010] The third subject of the present invention is video system comprising one or more high resolution network video cameras, computer and monitor, wherein said computer is configured to receive images from said video cameras by means of computer network and wherein said computer is configured to concurrently transmit different segments of said images to one or more remote users by means of computer network and wherein two or more of said remote users may request and receive distinctly different segments or sub-windows of said images.

[0014] Present invention discloses video surveillance system comprising previously disclosed by the author network cameras with massively parallel image processing, compression and network server.

[0015] The subject of the present invention is video surveillance system that comprises a number of high resolution network video cameras, wherein all or some of said cameras are capable of concurrent transmission of multiple image formats at video rates, common to these cameras video server/computer and system-level software, wherein said server is configured to acquire the images from said cameras, to store said images in digital archives, to display said images on one or more monitors and to respond to the requests of local and remote users by sending to said users requested images over local or wide area networks. FIG. 3 depicts block-diagram of video system based on high resolution network cameras.

[0016] The other subject of the present invention is video system comprising one or more high resolution network video cameras, computer and monitor, wherein said computer is configured to receive images from said video cameras by means of computer network and wherein said computer is configured to display said images on said computer monitor and wherein said display of said images comprises display mode wherein full field of view image and a sub-window of said image are displayed on said monitor simultaneously. FIG. 2 depicts an illustration of some of the operational modes of the system disclosed in the present invention.

[0017] The third subject of the present invention is video system comprising one or more high resolution network video cameras, computer and monitor, wherein said computer is configured to receive images from said video cameras by means of computer network and wherein said computer is configured to concurrently transmit different segments of said images to one or more remote users by means of computer network and wherein two or more of said remote users may request and receive distinctly different segments or sub-windows of said images. FIG. 1 depicts multi-user digital pan, tilt and zoom enabled by high-resolution network cameras.

[0018] In the preferred embodiment of the present invention, network cameras are based on image sensors capable of generating high resolution images at video rates, dedicated image processing pipeline capable of performing image processing at the speed of operation of the image sensor, image buffer memory for storing full resolution processed images and network interface with sufficient bandwidth to support video rate transmission of generated video.

[0019] In the preferred embodiment of the present invention, to reduce operation computational resources and network bandwidth, the cameras are capable of simultaneous generation and transmission of full field of view reduced-resolution (decimated) images along with the specified full resolution regions or sub-windows of the image. To achieve this capability, the cameras scan their image buffer memory repeatedly, forming requested images and transmitting them over network either using multiple logical connections or sequentially transmitting multiple image formats back-to-back over single logical connection.

[0020] In the preferred embodiment of the present invention multiple cameras are connected to a video server common to them, where said server represents a computer configured to execute system-level surveillance software and wherein said server/computer is also configured to perform real-time decompression of all or some of the incoming video streams and to display said video streams on high-resolution local monitor as well as on TV monitors connected to its output video device via appropriate modulators/interfaces.

[0021] In the preferred embodiment of the present invention, the surveillance system is configured to overlay full field of view reduced resolution decimated video images with full-resolution image sub-windows or rectangular image regions, wherein resulting composite image forms picture-in-picture (PIP) arrangement, wherein said composite image primarily consists of zoomed portion of full filed of view image with the second image overlaid in the corner of the main image. In the preferred embodiment of the present invention, said composite image is either displayed on the local monitor or modulated on the TV channel allowing simultaneous observation of high resolution (zoomed) portion of the image together with full field of view (possibly decimated) image on the same monitor. This mode of system operation avoids the shortcomings of low-resolution systems where the user has to choose between full field of view and zoomed imagery.

[0022] In one of the embodiments of the present invention, video server acquires from connected to it cameras complete full resolution images and processes them to produce picture-in-picture display image described above. In another embodiment of the present invention, the cameras simultaneously deliver full field of view images and full resolution image sub-windows. The potential advantage of this mode of operation may be illustrated by considering formation of NTSC image comprising 720×480 full resolution sub-window of full 1600×1200 camera image overlaid with 160×120 decimated full field of view image. If this image is formed on the server, then entire 1600×1200 image needs to be transmitted off camera. On the other hand only two images 640×480 and 160×120, respectively need to be transmitted concurrently off camera if the images can be formed in the camera, resulting in over 600% savings of network bandwidth.

[0023] In the preferred embodiment of the present invention the surveillance system is configured to support multiple user-selectable modes of image acquisition and display, wherein said modes comprise full resolution image display, simultaneous display of multiple decimated images acquired from different cameras on the same monitor, and picture-in-picture display of zoomed (windowed) and full-field-of-view video acquired from the same camera on the same monitor.

[0024] In one of the embodiments of the present invention three image formats are simultaneously transmitted off camera, wherein said image formats comprise reduced resolution decimated full field of view images, full resolution windowed images and an entire full resolution complete image, wherein first two image formats are transmitted at video frame rate for picture-in-picture display, while later format is transmitted at much slower rate, wherein full resolution complete image is stored in video archives and also provides the means for remote system users to gain independent access to pan, tilt and zoom capabilities, wherein said remote users request from the server and receive over the network of image sub-windows, wherein the system is configured to deliver to each remote user the images of specific size and resolution as requested by each said user.

[0025] To achieve real-time display of compressed video acquired from high-resolution network cameras the receiving computer first decompresses the acquired video streams. For sufficiently high camera resolution and frame rate, the update rate of the display may become slower than the frame-rate of said cameras due to CPU limitations and high computational complexity of said image decompression. In one of the embodiments of the present invention, the computer acquiring images from said high resolution cameras comprises an Ethernet network interface adapter (network card) wherein said network interface adapter comprises on-board real-time video decompression and direct memory access (DMA) engines. In this later embodiment said network adapter is configured to decompress acquired from said cameras images into bitmaps and to directly place said bitmaps in the computer memory, thus reducing CPU load.

[0026] In the preferred embodiment of the present invention said server/computer is also capable of servicing requests from local and remote users for the access to the specific parts of the system imagery. In the preferred embodiment, the requests from the remote user may specify resolution and size and position of image sub-window, that is then delivered to said user over local or wide area network. In the preferred embodiment of the present invention system server comprises the means for de-compressing, decimating and windowing acquired images, re-compressing said windowed or decimated images and sending said images to said remote users by means of standard network protocols such as TCP/IP. This capability allows multiple system users to gain simultaneous access to individual windows of interest by specifying their own pan, tilt and zoom (PTZ) settings of the camera. This presents a drastic contrast with optical zoom and mechanical pan and tilt wherein only one user can be in control of the camera at any given time.