High resolution video camera apparatus having two image sensors and signal processing

[0002] 1. Background—Field of the Invention

[0003] This invention relates to a novel high-resolution color video camera having high-resolution color image capture and, more particularly, a high-resolution color video camera having two different image sensors and a signal processing apparatus for generating a high-resolution color image.

[0004] 2. Background—Description of the Prior Art

[0005] Color video cameras are well known and used for many years in television production, news gathering and cinematography. Recent improvement in the sensors manufacturing process made possible the development of high-resolution color sensors with several million active pixels, and thus the development of high-resolution (high definition) video cameras, which are to replace the conventional standard definition television cameras and to enable the use of these new cameras in the motion picture industry. Parallel with that, new high definition video standards have been developed to facilitate this process.

[0006] Several types of solid-state video cameras utilizing a plurality of image sensors are already known. Prior art cameras, such as described in the U.S. Pat. Nos. 4,823,186 and 4,876,591 to Muramatsu, use a beam splitter and two image sensors—one monochrome and one color. The obtained video signal is a combination of two luminance signals, Y_Land Y_H, and two low frequency color difference signals, R_L-Y_Land B_L-Y_L. The monochrome imager generates only the high-resolution portion Y_Hof the luminance signal Y. The color imager generates the low-resolution portion Y_Lof the luminance signal and the two color difference signals R_L-Y_Land B_L-Y_L. The camera device is intended for an interlaced NTSC application, in which a video frame is captured every {fraction (1/30)} of a second ({fraction (1/60)} of a second for odd fields and {fraction (1/60)} of a second for even fields). In this mode of interlaced scanning an artifact, known as interline flicker, is present. This artifact is due to “Moire patterns”—high-resolution information near the Nyquist limit of resolution that produces a beat at low spatial frequencies resulting in flicker at 30 Hz (for NTSC standard signals). To avoid ‘Moire pattern’, Muramatsu advised us that the bandwidth for Y_L, R_L-Y_Land B_L-Y_Lsignals should be from 0 to 0.7 MHz, and the bandwidth for Y_Hshould be from 0.7 to 4.2 MHz. Unfortunately, these bandwidth restrictions limit the application of the camera device to the conventional NTSC standard. Being limited to NTSC standard, which is interlaced in its nature and has a bandwidth limit of 4.2 MHz, the method is not applicable to high-resolution machine vision and computer cameras, which predominantly use progressive scanning to eliminate ‘Moire pattern’. Furthermore, this camera is not compatible with newly developed digital and high definition television (HDTV) standards, which also use progressive scanning with 480 lines per frame, 720 lines per frame, and 1080 lines per frame. In addition, high definition luminance Y and the corresponding color difference signals C_Rand C_Bhave bandwidth much higher than NTSC. For example, for 720 lines per frame, the luminance bandwidth is about 30 MHz, and C_Rand C_Bbandwidth is 15 MHz each. Moreover, the low frequency luminance signal Y_Lis derived from the color imager, so an imager with tri-color filters (red, green, blue or yellow, cyan, magenta) is required. Thus, the implementation of an imager with only two types of color filters (such as red and blue) is not possible.

[0007] In U.S. Pat. No. 5,978,023 to Glenn, and in the article ‘The Development of a 1920×1080 Pixel Color CCD Camera Progressively Scanned at 60 Frames per Second’, by Glenn and Marcenka, a video camera is described, where together with line and pixel interpolation the camera uses high-pass filtering and frame interpolation on the luminance channel for increasing the frame rate. As a result a high-resolution 1920×1080 image scanned at 60 frames per second is generated. The output data rate of this image is about 3.3 Giga bits per second, which is beyond the capacity of the current analog or digital data recorders, editing and transmitting systems. In the majority of applications this increasing of the frame rate is not justified, does not improve the picture quality, and unnecessarily overloads and increases the computational time. In addition, particularly for digital cinema applications image scanning is only at 24 frames per second. In some special effects fast motion capture is necessary, but 60 frames per second is not fast enough and further complicates the recording, because the existing video recorders can record either normal speed at 24 frames per second or fast capture at three times the speed, i.e. 72 frames per second. In addition, this fast capture mode is used for selected frames only, and not for the entire production.

[0008] Accordingly, it is a general object of this invention to provide a novel apparatus for high-resolution and high frame rate image capture that can operate at the newly developed high definition television and digital cinema standards. It is a further object of the present invention to provide an improved progressively scanned high-resolution color camera apparatus, which is compatible with the existing data recording and editing equipment. It is also a further object of tile present invention to provide an improved progressively scanned high-resolution color camera apparatus and method for machine vision and computer applications.

[0009] These and other objects, advantages and features of the present invention will become more obvious from the following detailed description of the preferred embodiments along with the accompanying drawings.

[0010] Accordingly, the present invention provides a novel high-resolution video camera able to capture high-resolution images with high frame rate, where the video signal is produced using one monochrome and one color image sensors, and a signal processing apparatus. This novel apparatus does not involve frame rate temporal interpolation and high-pass filtering in its work.

[0011] According to one preferred embodiment of the present invention, the high-resolution camera includes a camera lens, a beam splitter, and two image sensors—one color and one monochrome, optically aligned with the lens and the beam splitter. The optical beam received from the camera lens is equally split into two beams via a 50/50 beam splitter and each beam projects an image onto the corresponding image sensor. The beam splitter can have unequal dividing ratio, such as 60/40, 30/70 or others. Both image sensors have the same active area and the same number of pixels horizontally and vertically. If sensors with different active areas are used, an additional lens or set of lenses is used to alter one or both optical beams divergence angle, modifying the optical beam cross-section in such a way that the new image size corresponds to the image sensor's active area. Thus, regardless of their size, both sensors will capture exactly the same scene.

[0012] In this embodiment both image sensors are scanned synchronously using a progressive scanning mode. In this particular application the monochrome imager has 1920 (M_H=1920) horizontal pixels per line and 1080 (M_v=1080) vertical lines. This imager is scanned at 24 frames per second. The output signal is digitized with 12-bit A/D and results in a high-resolution luminance signal Y_H.The resolution of this signal is 1920 (Y_HH=1920) horizontal pixels per line and 1080 (Y_HV=1080) vertical lines. An additional 12-to-10 bit gamma correction can be used.

[0013] The color imager has red, green and blue color filters arranged as RGBG vertical stripes, i.e. the first stripe has red color, the second—green, the third—blue, the fourth—green, the fifth is again red, an so on. This imager is scanned at 72 frames per second adding three lines at a time (3:1 binning), and generates three low-resolution color signals red, green and blue. The spatial resolution of the red and blue signals is 480 horizontal pixels per line and 360 vertical lines, and the spatial resolution of the green signal is 960 horizontal pixels per line and 360 vertical lines. These signals are digitized using a triple 12-bit A/D converter, and then converted to two low-resolution color difference signals, C_Rand C_B, and one low-resolution luminance signal Y_L. The spatial resolution of each color difference signal C_Ror C_Bin this particular application is 480 (C_H=480) horizontal pixels per line and 360 (C_V=360) vertical lines. The spatial resolution of the low-resolution luminance signal Y_Lis 960 (Y_LH=960) horizontal pixels per line and 360 (Y_LV=360) vertical lines. An additional 12-to-10 bit gamma correction can be used on each signal.

[0014] Both image sensors are scanned synchronously as described below. At frame 1 both image sensors are scanned, at frames 2 and 3 only the color sensor is scanned, at frame 4 both image sensors are scanned, at frames 5 and 6 only the color sensor is scanned, at frame 7 both image sensors are scanned, and so on. In this embodiment Y_H1, Y_H4, Y_H7and so on, represent the high-resolution luminance from frames 1,4,7 and so on, and C_R1, C_B1, Y_L1, C_R2, C_B2, Y_L2, C_R3, C_B3, Y_L3, and so on, represent the low-resolution luminance and color differences from frames 1,2,3 and so on. Thus a synchronous scanning at 24 frames per second for the monochrome imager and 72 frames per second for the color imager is achieved. In this mode of scanning each frame from the monochrome imager has a corresponding color frame.

[0015] The signals C_R, C_Band Y_Lare supplied to a signal processing module. First, the signals C_R1and C_B1are digitally interpolated 1:4 in horizontal direction and 1:3 in vertical direction to C_HR1and C_HB1, which have the same pixel resolution as the corresponding high-resolution luminance signals Y_H1. C_HR1, C_HB1and Y_H1are temporally aligned using an appropriate delay line and digitally mixed together. The same procedure is repeated for the frames 4, 7, etc. In this way a high-resolution image scanned at 24 frames per second is generated. On the next step the low-resolution luminance signals Y_L1, Y_L2and Y_L3, are selected, and two new motion vector signals M_Y2and M_Y3are generated using frame subtraction. M_Y2is generated when Y_L1is subtracted pixel by pixel from Y_L2, and M_Y3is generated when Y_L1is subtracted pixel by pixel from Y_L3. The same procedure is repeated for frames 4, 5, 6, for frames 7, 8, 9, and so on. The existing editing equipment can use this motion information to generate special motion effects, which will be added to the existing high-resolution image. For more precise motion an additional set of motion signals M_R2, M_R3and M_B2, M_B3can be derived in the same way from C_R1, C_R2, C_R3and C_B1, C_B2, C_B3.

[0016] The newly generated motion signals M_Y2, M_Y3, M_R2, M_R3, M_B2, and M_B3are mixed together with the newly generated high-resolution image forming in this way a composite data stream containing all image and motion information. Additional information such as date, time, camera location, etc. can be added to the data stream if needed. This composite data stream is further compressed if required to a 1.5 Giga bits per second data rate, and thus it can be recorded on the existing data recorders.

[0017] In some specific applications high frame rate information is not required, so both imagers can be scanned with the same frame rate.

[0018] The invention may take form in various parts and arrangements of parts or in various steps and arrangements. The drawings are only for purposes of illustrating a preferred embodiment and are not to be construed as limiting the invention.

[0019] FIG. 1 shows a block diagram of one form of the high-resolution color video camera.

[0020] FIG. 2 shows an alternative embodiment of the high-resolution color video camera of FIG. 1.

[0021] FIG. 1 shows a block diagram of one form of the high-resolution color video camera designed for high-resolution image capture. With reference to FIG. 1, a camera lens 101 is optically aligned with a beam splitter 102, a color image sensor 103, and a monochrome image sensor 104. The beam splitter 102 can be a half-silvered mirror, pellicle or similar device. In one preferred embodiment both image sensors 103 and 104 have the same active area, the same number of pixels per line—1920, and the same number of vertical lines—1080. The color image sensor 103 has a set of red (R), green (G) and blue (B) color filters placed on its surface, arranged as R G B G vertical stripes. Many other color combinations such as magenta, yellow, green and cyan; yellow and cyan; red and blue can be used. Both image sensors 103 and 104 are scanned synchronously using progressive scanning, where image sensor 104 is scanned at a relatively slow scan rate, and the image sensor 103 is scanned at a relatively fast frame rate.

[0022] The sensor 104 has output representative of the luminance level of a scene being imaged and can be any monochrome CCD or CMOS imager or other imaging device. The output of imager 104, which in this application is analog, is connected to the input of a first analog to digital converter 105, the output of which is connected to a first gamma corrector 106, which outputs a high-resolution luminance Y_H. If imager 104 generates a direct digital output, no analog-to-digital converter 105 is necessary. The sensor 103 has an output representative of the color pattern of the imaged scene, and can be any color CCD or CMOS imager or other imaging device. The output of 103, which in this application is analog, is connected to the input of a second analog to digital converter 107, the output of which is connected to a second gamma corrector 108. If imager 103 generates digital output, no analog-to-digital conversion is necessary.

[0023] The output of the gamma corrector 108 is connected to a signal processing apparatus 110, which first processing module comprises a color difference encoder 111. The encoder 111 has three outputs, which in the present embodiment are one low-resolution luminance signal Y_Land two color difference component signals C_Rand C_B. The output signals Y_L, C_Rand C_Bare coupled to a motion prediction module 114. In addition, the signals C_Rand C_Bare coupled to a dual pixel and line rate converter 112. The output signals C_HRand C_HBfrom the converter 112, along with the output Y_Hfrom the gamma corrector 106, are connected to the corresponding inputs of an adder 113. The corresponding output signals from 113 and 114 are connected to a data mixer 115.

[0024] Light received from the scene being imaged is focused by the camera lens 101, through the beam splitter 102 onto the front surfaces of the imager 104. Part of the light is reflected from the beam splitter 102 toward color imager 103, which receives a left-right reversed image. The beam splitter may distribute the light equally or unequally between said image sensors, where more light could be distributed towards the color imager 103. Both imagers 103 and 104 are optically aligned, so they receive exactly the same part of the scene, i.e. the top right pixel of imager 103 is optically aligned with the top left pixel of imager 104 and so forth (accounted for left-right image reversal). The image sensor 104 is scanned at a relatively slow scan rate, which in this embodiment is 24 frames per second, to produce 1080 visible lines per frame. The image sensor 103 is scanned faster than sensor 103, which in this embodiment is 72 frames per second by adding three lines at a time, to produce 360 visible lines per frame. At frame 1 both image sensors are scanned, at frames 2 and 3 only the color sensor 103 is scanned, at frame 4 both image sensors are scanned, at frames 5 and 6 only the color sensor 103 is scanned, at frame 7 both image sensors are scanned, and so on. 24 frames per second progressive scanning for imager 104, and 72 frames per second progressive scanning for imager 103 should not be considered as a limitation. Other frame rates, as well as interlaced scanning can be used.

[0025] The imager 104 creates a high frequency luminance signal, which carries complete information about the scene gray level pattern. Converter 105 digitizes the output signal from imager 104, and the digital data stream is gamma corrected using gamma corrector 106. In this embodiment the converter 105 is a 12-bit A/D converter, the corrector 106 compresses the data to 10 bits, and the high-luminance Y_Hhas a resolution of M_V=1080 vertical lines and M_H=1920 horizontal pixels per line.

[0026] The color imager 103 creates a set of three signals, providing complete information about the scene color pattern. The format of said color signals depends on the actual imager, and in this particular application the signals are carrying information about green, red and blue contents of the scene. Since the imager 103 receives a mirror image of the scene (left-right reversal), the imager 103 is scanned in reversed horizontal order from right to left, i.e. for each horizontal line the last pixel is clocked out first, and the first pixel is clocked last. This line reversal is not to be limited only to a hardware implementation, in some embodiments this can be done in software. Converter 107 digitizes the output signal from imager 103, and the digital data stream is further gamma corrected using gamma corrector 108. In this embodiment the converter 107 is a 12-bit A/D converter, and the corrector 108 compresses the data to 10 bits. The gamma-corrected data stream from 108 is converted in encoder 111 to a set of one low-resolution luminance signal Y_Land two color difference component signals C_Rand C_B. The spatial resolution of the luminance signal Y_Lin present embodiment corresponds to 360 vertical lines and 960 horizontal pixels per line, and the resolution of each color difference signal C_Ror C_Bis 480 horizontal pixels per line, and 360 vertical lines.

[0027] The converter 112 processes only the color difference signals C_Rand C_Bfrom frames 1, 4, 7 and so on, i.e. only the color frames that have a matching mono frame. Pixel and line interpolation is a well-known prior art. In the present embodiment the converter 112 interpolates the pixels in horizontal direction and the lines in vertical direction. During the first step, converter 112 creates one additional pixel between every two existing adjacent pixels, based on the average signal level of said adjacent pixels, and thus doubling the number of pixels in horizontal direction. This process is repeated one more time where, based on the same averaging scheme, one additional pixel is created in horizontal direction between every two already interpolated pixels. This results in quadrupling the number of horizontal pixels. During the second step two additional lines are created between every two existing adjacent lines in vertical direction, based on the average signal level of said adjacent lines. In this way every high-resolution luminance pixel from the mono imager 104 will have two corresponding color difference pixels from imager 103. The interpolated color signals C_HRand C_HBand the luminance signal Y_Hare temporally aligned using, if necessary, an appropriate delay line, and then digitally mixed together in adder 113. The latter outputs a serial data stream representing the high-resolution image scanned at the slower rate, which in this embodiment is 24 frames per second. The overall image resolution will be determined by the resolution of the luminance signal Y_Hproduced by the monochrome imager 104.

[0028] Parallel with the interpolation process, the output signals Y_L, C_Rand C_Bfrom each frame are processed in the motion predictor 114. Module 114 derives two motion signals M_Y2and M_Y3, each generated when the luminance from frames 2 and 3 is subtracted pixel by pixel from frame 1. If luminance signals from frames 1, 2 and 3 are noted as Y_L1, Y_L2and Y_L3, so M_Y2=Y_L2−Y_L1, and MY₃=Y_L3−Y_L1. The same procedure is repeated for C_Rand C_B, and results in four motion signals M_R2, M_R3and M_B2, M_B3. The newly developed motion signals along with the high-resolution images produced from 113 are mixed in mixer 115 into a composite data stream, compressed if necessary, and formatted into a standard 20 bits wide 1.5 Giga bits per second data rate. Additional information such as date, time, camera location, etc. can be added to the data stream if needed. This composite data stream is now compatible with the existing data recording and editing equipment, and in the same time contains complete image resolution and motion information.

[0029] FIG. 2 shows an alternative embodiment, which is modified from and basically similar in structure to the embodiment shown in FIG. 1. In FIG. 2, like parts, which correspond to those in FIG. 1, are marked with the same reference numerals. In this modified design the camera lens 101 is optically aligned with the beam splitter 102, monochrome image sensor 104, and a color image sensor 201. In this embodiment both image sensors 201 and 104 have the same active area, the same number of pixels per line—1920, and the same number of vertical lines—1080. The color image sensor 201 has a set of red (R), green (G) and blue (B) color filters placed on its surface, arranged in R G B G square, known as “BAYER” pattern. Many other color combinations such as magenta, yellow, green and cyan; yellow and cyan; red and blue can be used. In addition, the color image sensor 201 may be a combination of a monochrome image sensor 104 and a revolving color wheel, the latter comprising a set of color filters described above, wherein the rotation of said color wheel is synchronized with the scanning mode.

[0030] Both image sensors 201 and 104 are scanned synchronously using progressive scanning at the same frame rate. The output of imager 104, which in this application is analog, is connected to the input of the first analog to digital converter 105, the output of which is connected to the first gamma corrector 106, which outputs a high-resolution luminance Y_H. If imager 104 generates a direct digital output, no analog-to-digital converter 105 is necessary. The sensor 201 has an output representative of the color pattern of the imaged scene, and can be any color CCD or CMOS imager or other imaging device. The output of 201, which in this application is analog, is connected to the input of a second analog to digital converter 107, the output of which is connected to a second gamma corrector 108. If imager 201 generates digital output, no analog-to-digital conversion is necessary.

[0031] The output of the gamma corrector 108 is connected to a signal processing apparatus 210, which first processing module comprises a color difference encoder 211, which has two outputs-color difference component signals C_Rand C_B. The output signals C_Rand C_Bare coupled to a dual pixel and line rate converter 112. The output signals C_HRand C_HBfrom the converter 112, along with the output Y_Hfrom the gamma corrector 106, are connected to the corresponding inputs of an adder 213.

[0032] The embodiment according to FIG. 2 works in the same way as described in FIG. 1. Light received from the scene being imaged is focused by the camera lens 101, through the beam splitter 102 onto the front surfaces of the imager 104. Half of the focused light is reflected from the beam splitter 102 toward color imager 201, which receives a left-right reversed image. Both imagers 201 and 104 are scanned at 30 frames per second. 30 frames per second progressive scanning should not be considered as limitation. Other frame rates, as well as interlaced scanning can be used.

[0033] The imager 104 creates a high frequency luminance signal, which carries complete information about the scene gray level pattern. Converter 105 digitizes the output signal from imager 104, and the digital data stream is gamma corrected using gamma corrector 106. In this embodiment the converter 105 is a 12-bit A/D converter, the corrector 106 compresses the data to 10 bits, and the high-luminance Y_Hhas a resolution of M_V=1080 vertical lines and M_H=1920 horizontal pixels per line.

[0034] The color imager 201 creates a set of three signals, providing complete information about green, red and blue contents of the scene. Since the imager 201 receives a mirror image of the scene (left-right reversal), the imager 201 is scanned in reversed horizontal order from right to left, i.e. for each horizontal line the last pixel is clocked out first, and the first pixel is clocked last. Converter 107 digitizes the output signal from imager 103, and the digital data stream is further gamma corrected using gamma corrector 108. In this embodiment the converter 107 is a 12-bit A/D converter, and the corrector 108 compresses the data to 10 bits. The gamma-corrected data stream from 108 is converted in encoder 211 to a set of two color difference component signals C_Rand C_Beach with resolution 480 horizontal pixels per line, and 540 vertical lines.

[0035] The converter 112 processes only the color difference signals C_Rand C_B. During the first step, converter 112 creates one additional pixel between every two existing adjacent pixels, based on the average signal level of said adjacent pixels, and thus doubling the number of pixels in horizontal direction. This process is repeated one more time where, based on the same averaging scheme, one additional pixel is created in horizontal direction between every two already interpolated pixels. This results in quadrupling the number of horizontal pixels. During the second step one additional line is created between every two existing adjacent lines in vertical direction, based on the average signal level of said adjacent lines. In this way every high-resolution luminance pixel from the mono imager 104 will have two corresponding color difference pixels from imager 201. The interpolated color signals C_HRand C_HBand the luminance signal Y_Hare supplied to the adder 213, where they are temporally aligned, if required, using an appropriate delay line, compressed if necessary, and formatted into a standard 20 bits wide 1.5 Giga bits per second data rate. Thus, the adder 213 outputs a serial data stream representing the high-resolution image. This composite data stream is now compatible with the existing data recording and editing equipment, and in the same time contains complete image resolution and motion information. Additional information such as date, time, camera location, etc. can be added to the data stream if needed.

[0036] Accordingly, the reader will see that the present invention provides novel apparatus and method for high resolution image capture that can operate at the newly developed high definition imaging television and motion picture standards. This novel apparatus does not involve high-pass filtering and frame rate interpolation in its work. The principles of the invention have been described with reference to the present embodiments. Obviously, many modifications and variations could occur to someone skilled in the art upon reading and understanding the preceding detailed descriptions. It is intended that the invention be construed as including all such modifications and alterations insofar as they come within the scope of the appended claims or equivalents thereof.