US 5,402,171 AGrant
Electronic Still Camera With Improved Picture Resolution by Image Shifting in a Parallelogram Arrangement
Issue Date:1995-03-28
•32 Claims
•37 Drawing Sheets
Abstract
An electronic still camera comprises a solid state image sensor for outputting a color picture signal corresponding an incident optical image, a signal processor for signal-processing the color picture signal from the solid state image sensor to produce a color still picture signal, a recorder for recording the color still picture signal produced by the signal processor, and a driver for moving at least one of the optical image and the solid state image sensor from an original position in a horizontal direction by a pixel pitch and in an oblique axial direction defined by horizontal and vertical lines extending respectively from the original position in the horizontal and vertical directions by half the pixel pitch.
Metadata
Assignee
- Kabushiki Kaisha Toshiba
Inventors
- Yoshitomo Tagami
- Yuji Ide
- Masafumi Umeda
- Toshihiro Morohoshi
- Kazuhiro Takashima
- Mitsuo Sasuga
- Akihiko Sugikawa
Application Information
Application Number:US 1185810
Filing Date:1993-09-10
Priority Date:1992-09-11
Art Unit:262
Classifications
IPC:
H04N 5335
Field of Search:
348219;332;335;322;271
Patent Drawings (37 sheets)
Description
Background of the Invention
1. Field of the Invention
The present invention relates to an electronic still camera.
2. Description of the Related Art
Electronic still cameras are classified into two types: the first type involves analog recording using a floppy disk and the second type involves digital recording using a semiconductor memory (RAM etc.). Digital recording has various advantages over analog recording. For instance, digital recording produces no image deterioration due to copying, and does not involve any rotating mechanism for image recording, which contributes to making the camera compact.
The solid state image sensors which are employed in ordinary commercial cameras typically have about 400,000 pixels in view of the cost of the solid state image sensors. The solid state image sensors having pixels of that quantity can reproduce nearly satisfactory pictures when viewed on existing monitors, but the reproduced pictures do not have a satisfactory quality when output on a HD (High Definition) monitor or hard-copied. A Hi-vision solid state image sensor (having about 200 million pixels) for high definition is expensive and is not therefore suitable for commercial electronic still cameras.
There is a still picture input system which uses an inexpensive solid state image sensor having about several hundred thousand pixels and accomplishes high resolution by shifting the solid state image sensor. According to this system, the red (R) filter of a color filter disk is located in front of the solid state image sensor and a picture signal is read from the solid state image sensor. This signal is subjected to a predetermined process in a pre-processor, and is then converted into a digital value by an A/D converter. The picture signal is then processed to have a recording format by a digital signal processor before being recorded on a recording medium. When R data of one frame is recorded by the above operation, the solid state image sensor is moved by a fine distance to obtain a high resolution, and then the recording operation similar to the above is performed to obtain a plurality of frame data for R. When processing for R is completed, the color filter disk rotates to put the green (G) filter in front of the solid state image sensor. The G signal is processed and recorded in the same manner as the R signal The same processing is likewise executed for the blue (B) filter.
A still picture with high resolution is acquired in the above manner. Since, however, the solid state image sensor used in this system is a monochromatic device and requires a rotary filter for picking up image information in a field sequential image pickup system, the camera body is large and the sensor can pick up the image of only a still object. In this respect this solid state image sensor is not suitable for an electronic still camera which picks up a moving object as a still picture.
When a black and white object such as characters is imaged by an electronic still camera using an ordinary solid state image sensor having about 400,000 pixels, color moire occurs due to the color filter array and a clear picture cannot be obtained. To reduce the color moire, a crystal optical filter is generally provided. But this method deteriorates the resolution and is undesirable.
Digital electronic still camera systems, which records a still picture as a digital signal in a card type semiconductor memory as a recording medium, have been announced or put on the market by several companies. This semiconductor memory card is detachably mounted into the camera body. The unit price for semiconductor memories per bit is still expensive and the memory capacity is still insufficient, so that an optical disk, magneto optical disk, hard disk, magnetic tape, magnetic bubble memory, etc. are used or usable to record data for a long period of time or record a vast amount of data. Imaged data can be recorded on the semiconductor memory card as well as the secondary recording medium for each still picture.
Those electronic still cameras can transfer digital picture data to the monitor of a personal computer or a work station and display thereon. The digital picture data is typically converted into an analog signal input to the RF terminal or video terminal of an ordinary monitor of the TV standards and displayed on the monitor. In this case, the reproduced images are those images which are formed on the imaging area of the image sensor at the imaging time, and this area is fixed.
The auxiliary recording medium such as a semiconductor memory card, disk, and tape can record imaging data for each still picture. One cannot see, at a glance, the contents of the digital system, such as a memory card, and the above video floppy system, unlike the conventional photographic print. To see the contents, the user should place the recording medium in a reproducing apparatus and reproduce it, or write a memo on the package.
Jpn. Pat. Appln. No. 2-234492, entitled "Digital Electronic Still Camera System" discloses as an image retrieval method, a technique of using the conditions at the imaging time to improve the efficiency of image retrieval. The imaging conditions include the date of the imaging, white balance, the amount of incident light, focus, aperture, zoom, the use or non-usage of flash, humidity, atmospheric pressure, ID of the imaging camera and the type of the lens. On the reproducing side, the imaging conditions are indicated in the form of questions to allow a retriever to answer the questions, or the retriever should input the imaging conditions himself or herself. For instance, probable questions ask if the imaging was done in a room or outside the room, and if the object is a person, scenery or printed matter. The reproducing section read the imaging condition data required by the questions performs the necessary operations and reproduces images from the one with the highest probability.
Electronic still cameras, which digitize analog image information acquired from an image sensor and then compresses the digital data before recording, have already been commercialized.
The zoom function is widely used in home video movie cameras and compact cameras. There are two zooming methods for home video movie cameras: the first method uses a zoom lens and the second one is electronic zooming that performs operations on picture signals output from the image sensor. While the zoom lens can acquire good images at the time of zooming, a larger lens is needed for higher magnification, which stands in the way of making cameras compact. The electronic zooming, on the other hand, does not require a zoom lens and can thus contribute to accomplishing light and compact video movie cameras.
In performing an electronic zooming process on an image, as the image is spatially sampled by the image sensor, interpolation between sample pixels should be performed in accordance with the magnification. In this case, a typical method is to prepare data of pixels to be interpolated through interpolation using actual pixel data acquired by the image sensor. To execute pixel interpolation in real time, a bi-linear interpolation which requires a small circuit scale is widely used.
An example of this bi-linear interpolation is described in, for example, IEEE Transactions on Consumer Electronics Vol. 37, No. 3, August 1991, "An Electronic Zoom Video Camera using Image scanner control." This zooming however impairs the band of the original signals, thus deteriorating the resolution. The existing video movie cameras output or record image information as analog signals, not as digital image data in compressed form.
With regard to image compression, compression using block coding is employed as the international standard for motion pictures and still pictures and the adaptation of this system to a Hi-vision still picture storage device, a TV telephone, TV conference system and so on is studied. In conducting electronic zooming on those stored or coded digital images on the reproduction side,-block deformation originated from the block coding would be sensed by human eyes, degrading the image quality.
Summary of the Invention
It is therefore a primary object of the present invention to provide a high-quality electronic still camera that uses a color solid state image sensor, which has conventionally been used in commercial systems, to acquire pictures which can be viewed on an HD TV and can provide satisfactory images when hard copied.
It is the second object of this invention to provide an electronic still camera that uses a color solid state image sensor, which has conventionally been used in commercial systems, to acquire high-quality pictures of a black and white object, such as characters, with high resolution and without color moire.
It is the third object of this invention to provide an electronic still camera which has a wider output range of reproduced images than the standard TV monitors and allows an operator to alter the image output range.
It is the fourth object of this invention to provide an electronic still camera which can efficiently retrieve a desired image from among recorded images.
It is the fifth object of this invention to provide an image sensor which has an electronic zoom function and a highly-efficient image compressing function with suppressed image-quality deterioration even through those functions involve processing that normally deteriorate the image quality.
To achieve the first object, an electronic still camera according to this invention comprises a solid state image sensor having a color filter including a plurality of color filter elements indicating a plurality of colors which are arranged in horizontal and vertical axial directions, and a plurality of pixels at positions corresponding to the color filter elements, for outputting a color picture signal corresponding to an incident optical image, a moving unit for moving at least one of the optical image and the solid state image sensor from an original position in at least two directions by a pixel pitch between the pixels and half the pixel pitch, respectively, one of the directions being one of the horizontal and vertical axial directions and the other an oblique axial direction (vector direction) defined by horizontal and vertical lines extending respectively from the original position in the horizontal and vertical axial directions, and a signal processing unit for signal-processing the color picture signal obtained from the solid state image sensor on the original position and at least one moved position, to produce a color picture signal of a color still picture.
The color solid state image sensor includes a color filter in which plural units of mosaic filters, each unit arranged in two rows and two columns, are cyclically arranged in the horizontal axial direction and a direction perpendicular thereto.
The one unit of a mosaic filter includes two first color filter elements F1, one second color filter element F2 and one third color filter element F3. The moving section shifts the solid state image sensor by a pixel pitch (Px) in the horizontal direction or by a pixel pitch (Py) in the vertical direction so that the first color filter elements F1 respectively come to the positions of the second and third color filter elements F2 and F3 and further shifts the color solid state image sensor obliquely by distances of (-1/2Px+1/2Py) and (1/2Px+1/2Py) or (1/2Px-1/2Py) and (1/2Px+1/2Py) so that the first color filter elements F1 are equivalently arranged in a checkered pattern with half a horizontal pixel pitch and half a vertical pixel pitch. Accordingly, data is picked up, at a total of four positions to acquire a color still picture.
Alternatively, the solid state image sensor may include a color filter in which plural units of mosaic filters, each unit arranged in two rows and four columns or four rows and two columns are cyclically arranged in the horizontal axial direction and a direction perpendicular thereto. The one unit of a mosaic filter includes four first color filter elements F1, two second color filter elements F2 and two third color filter elements F3. In addition to the shifting to four positions by the aforementioned method, additional shifting is performed as follows. With an offset given to the position to which the color solid state image sensor is shifted vertically by a vertical pixel pitch (Py) or horizontally by a horizontal pixel pitch (Px), shifting is likewise conducted to four positions from that offset position, so that the second and third color filter elements F2 and F3 are arranged in checkered patterns with half a horizontal pixel pitch and half a vertical pixel pitch. Accordingly, data is picked up at a total of eight positions to acquire a color still picture.
The first color filter element F1 is one of a white filter (W), a color filter (Y) having a luminance spectra transmission characteristic and a green filter (G), and the second and third color filter elements F2 and F3 are two of a yellow filter (Ye), a cyan filter (Cy), a red filter (R) and a blue filter (B).
Further, a solid state image sensor, which has color filter elements of a yellow filter (Ye) and a cyan filter (cy) for odd lines (or even lines), and a magenta filter (Mg) and a green filter (G) for even lines (or odd lines), is shifted horizontally by a pixel pitch (Px) and obliquely by (1/2Px+1/2Py) and (-1/2Px+1/2Py) to acquire data at a total of four positions in order to yield a color still picture. This solid state image sensor may be shifted horizontally by a pixel pitch (Px) and obliquely by (1/2Px+Py) and (-1/2Px+Py) to acquire data at a total of four positions in order to yield a color still picture. Furthermore, data may be obtained at eight positions to provide a color still picture.
Moreover, the red filter elements, green filter elements and blue filter elements of the color filter of the color solid state image sensor may be cyclically arranged in three columns, the color solid state image sensor may be shifted horizontally by 1.5Px, vertically by 1/2Py and obliquely by (1.5Px+1/2Py), and data is obtained at a total of four positions to provide a single color still picture. Alternately, the color solid state image sensor may be shifted horizontally by 1.5Px, and obliquely by (-3/4Px+1/2Py) and (3/4Px+1/2Py), and data is obtained at a total of four positions to provide a single color still picture.
The above-described color solid state image sensor has a charge draining function so that charges are drained during the shifting of the solid state image sensor by the moving section.
To achieve the second object of this invention, an electronic still camera according to this invention comprises a color solid state image sensor having a color filter; a color image signal processing section for performing signal processing on an output signal of the color solid state image sensor to produce a color picture signal of a still picture; a binary image signal processing section for performing binary processing on the output signal of the color solid state image sensor to produce a binary picture signal; and a recording section for recording the color picture signal acquired from the color image signal processing section or the binary picture signal acquired from the binary image signal processing section.
Alternatively, there is provided an electronic still camera comprising a first imaging module including at least an optical lens and a color solid state image sensor; a color image signal processor for performing signal processing on an output signal of the first imaging module to produce a color picture signal of a still picture; a second imaging module including at least an optical lens and a monochromatic solid state image sensor; a monochromatic image signal processing for performing signal processing on an output signal of the second imaging module to produce a monochromatic picture signal of a still picture; and a recording section for recording the color picture signal acquired from the color image signal processing section or the monochromatic picture signal acquired from the monochromatic image signal processing section, the first and second imaging modules being designed attachable and detachable.
To achieve the third object of this invention, there is provided an electronic still camera which has a normal imaging mode for performing single signal reading from an image sensor with a fixed reproduction image output range, and another mode for performing signal reading plural times by shifting an object forming position on the image sensor and performing signal processing suitable for this reading, thereby ensuring a variable image output range at the time of image reproduction.
The electronic still camera according to this invention comprises a solid state image sensor for forming the image of an object thereon; a moving section for moving the position of the object image on the solid state image sensor; a signal reading section for reading a still picture signal from the solid state image sensor plural times in accordance the movement of the position of the object image by the moving section; a recording section for recording that of the still picture signals read by the signal reading section which belongs to a designated predetermined area, on a recording medium; a reproducing section for reproducing a still picture signal recording on the recording medium; and a display section for displaying the still picture signal reproduced by the reproducing section.
Further, the reproduction image output range may be widened by using an image sensor having a wider image output range than the reproduction monitor.
To achieve the fourth object, according to this invention, there is provided an electronic still camera which comprises a camera section for recording picture signals, picked up by a solid state image sensor, on a first recording medium; and a reproducing section to which the first recording medium or a second recording medium on which plural pictures of picture signals recorded on the first recording medium are recorded is attachable and which reproduces a picture with the first or second recording medium attached thereto, the camera section having an imaging condition data recording section for, when recording a picture on the first recording medium, recording imaging condition data then at the same time, the reproducing section having a retrieval questioning section for setting questions for a picture to be output, and an output section for reading imaging condition data recorded on the first or second recording medium to determine a picture having a high probability being associated with question data from the retrieval setting section and outputting the determined picture with a high probability.
To achieve the fifth object, an image sensor of this invention comprises an interpolation section for providing electronically enlarged display of a predetermined region of a digital picture; and image compression means for performing image compression by block coding, interpolation being performed before image compression. Further, this image sensor is provided with a section for storing only that region to be enlarged by interpolation on a recording medium, when a high magnification is selected.
More specifically, this image sensor comprises a solid state image sensor; an interpolation section for performing interpolation on a still picture signal output from the solid state image sensor for image enlargement; a compression section for performing compression on the still picture signal interpolated by the interpolation section; a recording section for recording the still picture signal, compressed by the compression section, on a recording medium; a reproducing section for reproducing the still picture signal recorded on the recording medium; and a display section for displaying the still picture signal reproduced by the reproducing section.
The electronic still camera provided to achieve the first object uses a color solid state image sensor and moves the solid state image sensor or optical image in the horizontal axial direction, vertical axial direction and oblique axial direction in a manner suitable for the color filter array in HD mode to acquire data at a total of four positions, thereby providing a single still picture. This design can improve the resolution.
Further, as no charges are accumulated during the movement, the shift-oriented blur will nor occur so that a clear image can be obtained.
Furthermore, the signal processing is so designed as to eliminate the need for an additional signal processor for the HD mode, thus contributing to making the electronic still camera compact.
The electronic still camera provided to achieve the second object performs binary processing in a binary signal processor, putting importance on the resolution, when a color solid state image sensor is used to pick up an image that simply requires binary data, such as characters, thus allowing for effective reduction of data and yielding a high-resolution binary image.
As the imaging modules are designed to be attachable to and detachable from the camera body imaging matching the purpose is possible.
The electronic still camera provided to achieve the third object moves the position of the image sensor or moves the optical passage between the lens and the image sensor by use of means attaining the first object to shift the object image on the image sensor, and reads a picture signal at each position, thereby acquiring vertical scans twice the normal ones. A half of those vertical axial picture signals are processed as an effective picture portion and are recorded on a recording medium. The horizontal image angle of the image recorded on the recording medium is twice the normal angle.
In reproducing this image in the reproducing circuit, the display range of the reproduced image can be shifted horizontally by changing the horizontal read position in the read frame memory. The shifting the horizontal object image at the time of image input can improve the resolution.
Although an image sensor with a wide horizontal image angle is used and the center pixels are used in the normal imaging, the horizontal reproduction image range can be widened by using the entire pixels in another imaging mode.
The electronic still camera provided to achieve the fourth object records picture signals as well as position information, generated outside or in the electronic camera, together with other imaging condition data, on a recording medium at the imaging time. The image reproducing section reads the picture signals and the position information associated with the also recorded imaging condition data, and sequentially outputs pictures starting with the one having the position information that mostly likely matches with the retrieval question about the imaging position.
The electronic still camera provided to achieve the fifth object has a normal imaging mode and a zoom imaging mode. In normal imaging mode, this camera performs analog signal processing and analog-to-digital conversion on the signals from the image sensor, and then performs digital signal processing not including a zoom process before the signals are output or recorded.
In zoom imaging mode, the camera performs analog signal processing and analog-to-digital conversion on the signals from the image sensor, and then performs digital signal processing including a zoom process before the signals are output or recorded. The zoom processing section in zoom imaging mode selects different processes at different magnifications, the aforementioned low magnification and the aforementioned high magnification.
At the low magnification, the deterioration of the picture quality in image compression can be minimized by executing image compression after interpolation. At the high magnification, the influence of image compression to deteriorate the picture quality can be avoided by recording picture signals on a recording medium without compressing that portion alone which is to be enlarged by interpolation.
At either the low magnification or high magnification, therefore, the deterioration of the picture quality originated from image compression can be minimized and the amount of image data to be output on the recording medium can be reduced to allow a greater number of pictures to be recorded on a small-capacity recording medium.
Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out in the appended claims.
Brief Description of the Drawings
The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate presently preferred embodiments of the invention, and together with the general description given above and the detailed description of the preferred embodiments given below, serve to explain the principles of the invention.
FIG. 1 is a block diagram of an electronic still camera according to a first embodiment of the present invention;
FIG. 2 is a block diagram of an electronic still camera according to a second embodiment of the present invention;
FIG. 3 is a block diagram of an electronic still camera according to a third embodiment of the present invention;
FIG. 4 is a block diagram of an electronic still camera according to a fourth embodiment of the present invention;
FIGS. 5A and 5B are diagrams for respectively explaining a color filter array used in this invention and how the color filter array is moved;
FIG. 6 is a block diagram showing a digital signal processor in the electronic still camera of this invention;
FIG. 7 is a diagram showing the exemplary structure of a solid state image sensor;
FIGS. 8A through 8E are diagrams for explaining how electric charges are accumulated when the solid state image sensor in FIG. 7 is used in HD mode;
FIGS. 9A through 9C are diagrams for explaining another state of the accumulation of electric charges when the solid state image sensor in FIG. 7 is used in HD mode;
FIGS. 10A and 10B are diagrams illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 5A is used in HD mode;
FIG. 11 is a diagram for explaining the two-dimensional addresses in a buffer memory with the use of the equivalent color filter array in HD mode shown in FIGS. 10A and 10B;
FIGS. 12A and 12B are diagrams for respectively explaining a different color filter array used in this invention and how the color filter array is moved;
FIGS. 13A and 13B are diagrams illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 12A is used in HD mode;
FIGS. 14A and 14B are diagrams for respectively explaining another color filter array used in this invention and how the color filter array is moved;
FIGS. 15A and 15B are diagrams illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 14A is used in HD mode;
FIGS. 16A and 16B are diagrams for respectively explaining a further color filter array used in this invention and how the color filter array is moved;
FIG. 17 is a diagram illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 16A is used in HD mode;
FIGS. 18A and 18B are diagrams for respectively explaining the same color filter array as shown in FIG. 16A and the movement of that color filter array;
FIG. 19 is a diagram illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 18A is used in HD mode;
FIG. 20 is a diagram showing a color filter array of a color solid state image sensor used in the electronic still camera of this invention;
FIG. 21 is a diagram showing a different color filter array of a color solid state image sensor used in the electronic still camera of this invention;
FIG. 22 is a diagram showing another color filter array of a color solid state image sensor used in the electronic still camera of this invention;
FIG. 23 is a diagram showing a color filter array in which F1, F2 and F3 of the color filter array shown in FIG. 20 are a W (white) filter element, Ye (yellow) filter element, and Cy (cyan) filter element, respectively;
FIG. 24 is a diagram showing how a color solid state image sensor is shifted in HD mode of this invention;
FIGS. 25A and 25B are diagrams respectively illustrating equivalent color filter arrays when solid state image sensors with the color filter arrays shown in FIGS. 20 and 21 are used in HD mode shown in FIG. 24;
FIGS. 26A and 26B are diagrams illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIGS. 23 is used in HD mode shown in FIG. 24;
FIG. 27 is a diagram showing a different shifting state of a color solid state image sensor in HD mode of this invention;
FIGS. 28A and 28B are diagrams respectively illustrating equivalent color filter arrays when solid state image sensors with the color filter arrays shown in FIGS. 21 and 22 are used in HD mode shown in FIG. 27;
FIG. 29 is a diagram showing a color filter array adaptable to this invention in which F1, F2 and F3 of the color filter array shown in FIG. 20 are a G (green) filter element, R (red) filter element, and Cy (cyan) filter element, respectively;
FIG. 30 is a diagram showing a color filter array adaptable to this invention in which F1, F2 and F3 of the color filter array shown in FIG. 20 are a G (green) filter element, Ye (yellow) filter element, and Cy (cyan) filter element, respectively;
FIG. 31 is a diagram showing a color filter array adaptable to this invention in which F1, F2 and F3 of the color filter array shown in FIG. 20 are a Y (a color filter having a spectral transmission characteristic of a luminance signal) filter element, R (red) filter element, and Cy (cyan) filter element, respectively;
FIG. 32 is a diagram showing a color filter array of a color solid state image sensor for explaining another embodiment of this invention;
FIG. 33 is a diagram showing a color solid state image sensor having the color filter array shown in FIG. 32 is shifted in HD mode;
FIG. 34 is a diagram illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIGS. 32 is used in HD mode shown in FIG. 33;
FIG. 35 is a diagram showing a color solid state image sensor having the color filter array shown in FIG. 32 is shifted in HD mode;
FIG. 36 is a diagram showing a color solid state image sensor having the color filter array shown in FIG. 32 is shifted in HD mode shown in FIG. 35;
FIG. 37 is a diagram showing a different color filter array adaptable to this invention;
FIG. 38 is a diagram showing another color filter array adaptable to this invention;
FIG. 39 is a diagram showing a further color filter array adaptable to this invention;
FIG. 40 is a diagram showing the 8-position shifting state in HD mode of this invention;
FIG. 41 is a diagram showing another 8-position shifting state in HD mode of this invention;
FIG. 42 is a diagram showing an equivalent color filter array acquired when shifts as shown in FIGS. 40 and 41 are performed;
FIG. 43 is a diagram showing a different color filter array adaptable to the shifting shown in FIG. 40;
FIG. 44 is a diagram showing another color filter array adaptable to the shifting shown in FIG. 41;
FIG. 45 is a block diagram of an electronic still camera according to another embodiment, which is designed to prevent the occurrence of color moire;
FIG. 46 is a diagram showing a color filter array for explaining the embodiment shown in FIG. 45;
FIG. 47 is diagram illustrating an equivalent color filter array when a solid state image sensor with the color filter array shown in FIG. 46 is treated as a monochromatic device;
FIG. 48 is a block diagram of an electronic still camera according to a further embodiment associated with the embodiment shown in FIG. 45;
FIG. 49 is a diagram for explaining the functions of an electronic still camera according to a still another embodiment which can alter the display range;
FIG. 50 is a diagram for explaining the imaging range of an image in this embodiment;
FIG. 51 is a block diagram showing a reproduction signal processing system in this embodiment;
FIG. 52 is a diagram for explaining the necessary number of pixels in the image reproduction on a monitor in this embodiment;
FIG. 53 is a block diagram of a first example of an display memory shown in FIG. 53;
FIG. 54 is a block diagram of a second example of the display memory shown in FIG. 53;
FIG. 55 is a block diagram of a third example of the display memory shown in FIG. 53;
FIG. 56 is a block diagram of a fourth example of the display memory shown in FIG. 53;
FIG. 57 is a diagram of a first example of the address space of the display memory;
FIG. 58 is a diagram of a second example of the address space of the display memory;
FIG. 59 is a diagram of a second example of the address space of the display memory;
FIG. 60 is a diagram of an image sensor having an image angle longer in the horizontal direction than that of an ordinary image sensor;
FIG. 61 is a diagram exemplifying the reproduction screen showing the mode is a panorama mode;
FIG. 62 is a diagram showing an example of the reproduction screen indicating that the reproduction range is movable in both right and left directions;
FIG. 63 is a diagram showing an example of the reproduction screen indicating that the reproduction range is movable only in the left direction;
FIG. 64 is a block diagram showing a first example of a reproducing section equipped with a mark generator;
FIG. 65 is a block diagram showing a second example of the reproducing section equipped with a mark generator;
FIG. 66 is a diagram exemplifying an electronic still camera having an operation section equipped with a display section whose reproduction range is movable;
FIG. 67 is a diagram exemplifying the display in the finder of an electronic still camera having a panorama mode;
FIG. 68 is a diagram showing another example of the display in the finder of an electronic still camera having a panorama mode;
FIG. 69 is a block diagram of an output image memory section of an electronic still camera having top, bottom, right and left display ranges;
FIG. 70 is a diagram showing an example of an image display which represents that it has top, bottom, right and left display ranges;
FIG. 71 is a block diagram of the recording section of an electronic still camera according to a further embodiment, equipped with a retrieval function;
FIG. 72 is a diagram showing the case where the electronic still camera of this embodiment is connected to a GPS (Global Positioning System) receiver;
FIG. 73 is a block diagram showing the reproducing section of the electronic still camera according to this embodiment or the reproducing section of an electronic album having a medium for recording image data picked up by the electronic still camera;
FIG. 74 is a diagram showing the case where the electronic still camera according to this embodiment is used as a mobile telephone;
FIG. 75 is a block diagram of the electronic still camera of the embodiment shown in FIG. 71;
FIG. 76 is a diagram showing an example of a question screen for retrieving image data picked up by the electronic still camera of this invention;
FIG. 77 is a block diagram illustrating an operation for image retrieval;
FIG. 78 is a block diagram according to a still further embodiment which has a zooming function and a compressing function;
FIG. 79 is a flowchart illustrating the flow of the control according to this embodiment;
FIG. 80 is a diagram showing a compression system for blocked image data which is used in this embodiment;
FIGS. 81A through 81F are diagram for explaining the features of this blocked-image data compression system;
FIG. 82 is a diagram showing the band characteristic of a video signal and the band characteristic of a signal acquired by two-point interpolation;
FIG. 83 is a block diagram of an image sensor according to a different embodiment associated with FIG. 78;
FIG. 84 is a block diagram of an image sensor according to another embodiment;
FIG. 85 is a flowchart illustrating the flow of the control according to this embodiment;
FIG. 86 is a diagram showing an image format;
FIG. 87 is a diagram illustrating the principle of the function of a compression/extraction selecting section shown in FIG. 84;
FIG. 88 is a block diagram of an image sensor according to a further embodiment associated with FIG. 78;
FIG. 89 is a block diagram of an image sensor according to still another embodiment associated with FIG. 78; and
FIG. 90 is a block diagram of an image sensor according to a still further embodiment associated with FIG. 78.
Detailed Description of the Preferred Embodiments
FIG. 1 is a block diagram of an electronic still camera according to the first embodiment of the present invention. In this drawing, a color solid state image sensor such as a CCD (Charge Coupled Device) 1 is shifted by a piezo device 11 that is driven by a driver 2. The output terminal of the CCD 1 is connected to a buffer memory 6 sequentially via a pre-processing circuit 4 and an A/D converter 5 in series. The preprocessing circuit 4 and A/D converter 5 are controlled by sync signals output from a sync signal generator 3. The buffer memory 6 is controlled by a memory controller 7 which also receives a sync signal from the sync signal generator 3. The output terminal of the buffer memory 6 is connected via a signal processor 8 to a recording medium 9 such as a semiconductor memory. The output terminal of a mode selector 10 is connected to the sync signal generator 3.
The normal processing of the above electronic still camera will be described first.
In this case, the normal imaging mode is selected by the mode selector 10, and a sync signal pulse is sent to the driver 2 from the sync signal generator 3, allowing one frame of picture signals to be read from the driver 2. The driver 2 supplies a read signal to the color CCD 1 and supplies a drive signal to the piezo device 11 to shift the CCD 1. The picture signals read from the color CCD 1 are input to the pre-processing circuit 4 where it is subjected to a predetermined process such as pre-amplification, white balance or gamma correction. The signal from the pre-processing circuit 4 are converted into digital signals by the A/D converter 5, and the digital signals are then temporarily stored in the buffer memory 6. When one frame of picture data is stored in the buffer memory 6, it is processed in a non-interlace mode. If the readout from the color CCD 1 is interlaced data, the data can be processed in a non-interlace form by putting the picture signals through the buffer memory 6. The picture signals read from the buffer memory 6 are input to the signal processor 8. This signal processor 8 produces a luminance signal and a color difference signal from each picture signal, performs data compression and records the compressed data on the recording medium 9 such as a semiconductor memory.
The signal processing with the color filter elements of the color CCD 1 laid out as shown in FIG. 5A will be described with reference to a luminance/color difference signal processing circuit shown in FIG. 6.
The signal read from the buffer memory 6 is converted into a high band luminance signal Y.sub.H, a low band luminance signal Y.sub.L and color difference signals R-Y and B-Y by a matrix circuit 13. The following are the matrix equations for the G.sub.2(Y),3(X) pixels in FIG. 5A.
The matrix equations for the G.sub.3,3 pixels are as follows:
The color difference signals R-Y and B-Y passing through the matrix circuit 13, which performs the above processing, are subjected to band limitation by low-pass filters (LPF) 17 and 18. The luminance signal Y is produced by subjecting the Y.sub.L -Y.sub.H signal to band limitation by an LPF 15 and adding the signal Y.sub.H to the resultant signal. The thus produced luminance signal and the color difference signals are subjected to data compression by a compressing circuit (not shown) and the compressed signals are recorded on the recording medium 9.
A description will now be given of an HD recording mode.
FIG. 5B illustrates the shifting of the color CCD 1 in HD recording mode. First, the HD mode is selected by the mode selector 10, and four frames of sync signal pulses are output from the sync signal generator 3. Picture signals are read from the color CCD 1 first in the normal state (original position (1)), and are stored in the buffer memory 6. When the storage of one frame of picture signals is complete, the color CCD 1 is shifted horizontally from the original position (1) to the position (2) by Px.
Px and Py are horizontal and vertical pixel pitches, respectively. There may be various kinds of means to shift the color CCD 1. For instance, the color CCD 1 itself may be shifted by the piezo device 11 or the like, or a parallel-plate member 12 may be provided in front of the color CCD 1 as in the second embodiment shown in FIG. 2 and changed in its angle to change the optical passage, thereby ensuring equivalent shifting of the color CCD 1. Such shifting mechanisms are disclosed in Jpn. Pat. Appln. KOKAI Publication Nos. Sho 58-130677 and Sho 60-54576.
When the color CCD 1 is shifted to the position (2) from the position (1) by the above method, picture signals are read from the color CCD 1 and are then stored in the buffer memory 6. Likewise, picture signals are also read from the color CCD 1 at the position (3) (-1/2Px+1/2Py) and the position (4) (1/2Px+1/2Py) and are stored in the buffer memory 6. Each of the positions (3) and (4) is from the position (1) in a vector direction defined by horizontal and vertical lines extending respectively from the original position (1) in the horizontal and vertical axial directions, and the positions (1), (2), (3) and (4) correspond to four vertexes of parallelogram, respectively. In other words, the four lines connecting the positions (1), (2), (3) and (4) define a parallelogram.
Now the relationship between the operation of reading signals from the color CCD 1 and the pixel shift will be described in detail.
In the following description, an interline transfer CCT (IT-CCT) which can ensure full line scanning as shown in FIG. 7 is used as the color CCD 1. The CCD array has a vertical transfer section 402 provided adjacent to each row of photoelectric conversion elements 401 each consisting of a photodiode. The electric charges of the individual photoelectric conversion elements 401 are transferred to the associated vertical transfer section 403 and are then transferred to a horizontal transfer section 403 line by line by a field shift pulse .phi.V1. The electric charges are then output to an output terminal (OUT) from an output circuit 404 as electric signals by a pulse .phi.H. Provided at the other end of the vertical transfer section 402 is a charge drain section 405.
The state of the electric charges in the color solid state image sensor will be described referring to FIGS. 8A through 8E. In the case the processes at the positions (1), (2), (3) and (4) as shown in FIG. 5B are performed consecutively, the electric charges would be accumulated as shown in FIG. 8A. When the color CCD 1 is shifted, for example, from the position (1) to the position (2), the shifting takes time as shown in FIG. 8B, and the accumulation of the electric charges at the position (2) becomes inaccurate accordingly, yielding a blurred image. To overcome this problem, the electric charges are drained during the shifting as shown in FIG. 8C. By this charge draining, the accumulation of the electric charges after draining matches with the charge accumulation in a period excluding the shifting period as shown in FIG. 8D thus eliminating image blurring. The accumulated electric charges are transferred to the vertical transfer section 402, and are read therefrom with a delay of one frame period as shown in FIG. 8E.
Another example of shifting is to process picture signals at an interval of one frame as shown in FIGS. 9A to 9C. In this way the period of the accumulation of electric charges and the moving period can be made different from each other, preventing the occurrence of image blurring.
Although the present invention is applied to a full line scanning type IT-CCD in the foregoing description of this embodiment, this invention can be applied to all solid state image sensors having a charge draining function, such as an interlace scanning type IT-CCT and frame interline transfer type CCD (FIT-CCD).
FIGS. 10A and 10B illustrate an equivalent pixel arrangement when signal are read at the aforementioned four positions (1) to (4) corresponding to the four vertexes of parallelogram. It is apparent from those diagrams that the G filter elements are arranged in a checkered pattern with a horizontal pixel pitch of 1/2Px and a vertical pixel pitch of 1/2Py, and the R filter elements and B filter elements are alternately arranged at every third line at a vertical pixel pitch of 1/2Py. By properly selecting the arrangement of the color filter elements and the shifting direction in this manner, a shifted equivalent pixel arrangement can take a regular position, thus ensuring high definition (HD).
The signal processing with this pixel arrangement will now be described. The pixel arrangement shown in FIGS. 10A and 10B quite differs from the normal pixel arrangement, so that the signal processing for the former arrangement should also differ from that for the latter arrangement, thus requiring a separate signal processor. In this case, the circuit scale becomes large, which is undesirable particularly in making an electronic still camera compact The following will described how the signal processor in normal mode is used in the solid state image sensor in the case of FIGS. 10A and 10B.
First, frame data at four positions are stored in the buffer memory 6. In consideration of two-dimensional addresses, the frame data are written in the buffer memory 6 in the order shown in FIG. 11 by the memory controller 7. The array of the color filter elements of the CCD 1 (array A) is equivalent to the two-dimensional address arrangement with every two vertical lines of addresses treated as one line. The signal processing in this case is executed as follows.
For the vertical addresses with y numbered 1, 3, 5, . . . , processing is performed with respect to the horizontal addresses with x numbered 2, 4, 6, . . . , and for the vertical addresses with y numbered 2, 4, 6, . . . , processing is performed with respect to the horizontal addresses with x numbered 1, 3, 5, and so on
The computation for G.sub.3(y),6(x) pixels is performed as follows.
The operation for G.sub.4(y),5(x) pixels is performed as follows.
When the vertical addresses (y) are for the N-th line, a computation is performed using data of that line and the (N-2)-th line and data of the horizontal addresses (x) as apparent from the above. Accordingly, luminance signals can be produced with a horizontal pitch of 1/2Px, ensuring resolution more than twice the normal resolution. As an object in monochromatic mode is expressed only by the signals of the vertical lines, the vertical resolution becomes two times the normal resolution.
Through the above-described processing, a picture quality as high as that of an HD TV can be obtained. It is apparent from the comparison with the equations (1) to (8) that this embodiment performs the same computation as in the normal mode to execute the signal processing, so that the very same signal processor 8 can be used with a change in signal reading from the buffer memory 6.
Although the array A shown in FIG. 5A has been explained in the foregoing description, the same processing can be performed for an array B shown in FIG. 12A and an array C shown in FIG. 14A. FIGS. 13A and 13B and FIGS. 15A and 15B show equivalent color filter element arrays when the color CCD 1 is shifted horizontally from a position (1) to a position (2) by Px and obliquely from the position (1) to positions (3) and (4) by -1/2Px+1/2Py and 1/2Px+1/2Py as shown in FIGS. 5B, 12B and 14B as in the case of the array A in FIG. 5A. It is apparent that those color filter element arrays become quite the same as the array A in FIG. 5A. The array is not limited to those arrays and other arrays can be used to achieve the above-described embodiment simply by vertically arranging G filters in every line and vertically arranging R and B filters in every other lines.
A description will now be given of an array D (RGB stripe filter array shown in FIG. 16A. In this case, the color CCD 1 is shifted as shown in FIG. 16B. That is, the color CCD 1 is shifted horizontally from a position (1) to a position (2) only by 1.5Px, vertically therefrom to a position (3) by 1/2Py and obliquely therefrom to a position (4) by 1.5Px+1/2Py. Each of the positions (3) and (4) is from the position (1) in a vector direction defined by horizontal and vertical lines extending respectively from the original position (1) in the horizontal and vertical axial directions, and the positions (1) to (4) correspond to the four vertexes of parallelogram, respectively. It is apparent from the equivalent color filter element array (FIG. 17) obtained by the shift that the array becomes an RGB stripe array, so that the same signal processor 8 as used in the previous embodiment is used and the horizontal and vertical resolutions become twice the normal ones.
A description will now be given of the system of further improving the horizontal resolution for the array D shown in FIG. 18A. As shown in FIG. 18B, the color CCD 1 is shifted horizontally from a position (1) to a position (2) by 1.5Px, obliquely therefrom to a position (3) by -3/4Px+1/2Py, and obliquely therefrom to a position (4) by 3/4Px+1/2Py. The positions (1) to (4) correspond to the vertexes of parallelogram, respectively. As apparent from the equivalent color filter element array (FIG. 19) obtained by shifting the color CCD 1 as shown in FIG. 18B, further improvement on the horizontal resolution can be expected in this embodiment.
With the use of the buffer memory 6, the following system may be accomplished. As the capacity of the buffer memory 6 should be as large as four times the pixels of the color CCD 1 for the HD mode, the electronic still camera of the above-described embodiment can be used for rapid continuous shots in normal imaging mode because of the use of this buffer memory 6.
In the third embodiment shown in FIG. 3, consecutive picture signals from the color CCD 1 are consecutively stored in memory regions BM1 BM2 BM3 and BM4 in the buffer memory 6. Four consecutive frames of image data stored in the buffer memory 6 are read out from the BM1, and are subjected to digital signal processing before being recorded on the recording medium 9.
Likewise, the image data stored in the memory regions BM2, BM3 and BM4 are processed in the same manner as the image data stored in the memory region BM1. The recording time of a single frame of a picture is restricted by the signal processing time or the time for data recording of the recording medium 9 or the like, so that high-speed continuous shots are difficult. As a solution to this, several frames of pictures are written in the buffer memory 6 at a high speed, and are then subjected to signal processing before being recorded on the recording medium 9. This way, the restriction such as the signal processing time would be removed, thus ensuring fast continuous shots.
Although the foregoing description of the third embodiment has been given with reference to the case where the buffer memory 6 is used for the HD mode, a system without the buffer memory 6 may be accomplished as a different embodiment.
With reference to FIG. 4, this system will be described as the fourth embodiment.
First, picture signals are read out with the color CCD 1 positioned at a position (1) as shown in FIG. 5B, for example. The read signals are subjected to a predetermined process in the pre-processing circuit 4 and are then converted into digital signals by the A/D converter 5. The digital signals are recorded on the recording medium 9 without going through the signal processor 8 with the color filter element array unchanged. The same processing is performed when the CCD 1 is shifted to each of the positions (2), (3) and (4). When imaged data at the four positions are recorded on the recording medium 9, video signals are produced by the reproducing apparatus. In this manner, recording in HD mode can be executed without using the buffer memory 6.
A description will now be given of another example of signal processing in the embodiment shown in FIG. 2 with reference to the signal processor shown in FIG. 6, the color filter element array patterns shown in FIGS. 20 to 22 as well as FIGS. 23 through 26.
In the digital signal processor shown in FIG. 6, the signal read from the buffer memory 6 is converted into a high band luminance signal Y.sub.H, a low band luminance signal Y.sub.L and color difference signals R-Y and B-Y by the matrix circuit 13. The matrix equations performed in this matrix circuit 13 are expressed as follows with respect to the W.sub.2(Y),3(Y) pixels in FIG. 23.
The matrix equations with respect to the W.sub.3(Y),3(Y) pixels are expressed as follows.
SW.sub.y,x, SYe.sub.y,x and SCy.sub.y,x express output signals with the horizontal address (X) and vertical address (Y), With S.sub.W =R+G+B, S.sub.Ye =R+G and S.sub.Cy =G+B, R, G and B are normalized to "1." .alpha..sub.1, .alpha..sub.2, .alpha..sub.3, .beta..sub.1, .beta..sub.2 and .beta..sub.3 are matrix coefficients.
The color difference signals R-Y and B-Y passing through the matrix circuit 13, which performs the above processing, are subjected to band limitation by the low-pass filters (LPF) 17 and 18. The luminance signal Y is produced by subjecting the Y.sub.L -Y.sub.H signal to band limitation by the LPF 15 and adding the signal Y.sub.H to the resultant signal. The thus produced luminance signal and the color difference signals are subjected to data compression by a compressing circuit (not shown) and the compressed signals are recorded on the recording medium 9.
A description will now be given of an HD recording mode.
FIG. 24 illustrates the shifting of the color CCD 1 in HD recording mode. First, the HD mode is selected by the mode selector 10, and four frames of sync signal pulses are output from the sync signal generator 3.. Picture signals are read from the color CCD 1 first in the normal state (original position (1)), and are written in the buffer memory 6. When one frame of picture signals are written in the buffer memory 6, the color CCD 1 is shifted horizontally from the original position (1) to the position (2) by Px. Px and Py are horizontal and vertical pixel pitches, respectively. The shifting means is realized by means for shifting the CCD 1 or means for changing a optical path for the optical image as described in the first embodiment.
When the color CCD 1 is shifted to the position (2) by the above method, picture signals are read from the color CCD 1 and are then written in the buffer memory 6. Likewise, picture signals are also read from the color CCD 1 at the position (3) (-1/2Px+1/2Py) and the position (4) (1/2Px+1/2Py) and are written in the buffer memory 6. The picture signals acquired at the four positions in this manner equivalently become signal acquired from the pixels arranged as shown in FIGS. 26A and 26B. The equivalent array in the case where this method is applied to the color filter shown in FIG. 20 is illustrated in FIGS. 25A and 25B. FIG. 25A illustrates an equivalent array of the filter elements F1, and FIG. 25B an equivalent array of the filter elements F2 and F3.
It is apparent from those drawings that all the color filter elements are arranged at a pixel pitch of 1/2Py in the horizontal direction and the color filter elements w are arranged at a pixel pitch of 1/2Py in the vertical direction. The Ye and Cy components are alternately arranged at every third line at a pixel pitch of 1/2Py.
An example of the signal processing for the above array will be described below.
For the vertical addresses with y numbered 1, 3, 5, . . . , processing is performed with respect to the horizontal addresses with x numbered 1, 3, 5, . . . , and for the vertical addresses with y numbered 2, 4, 6, . . . , processing is performed with respect to the horizontal addresses with x numbered 2, 4, 6, and so on.
For example, the computation for G.sub.5(y),3(x) pixels is performed as follows.
The matrix equations with respect to the W.sub.4(Y),4(Y) pixels are expressed as follows.
SW.sub.y,x, SYe.sub.y,x and SCy.sub.y,x express output signals with the horizontal address (X) and vertical address (Y) in an equivalent array after shifting.
When the vertical addresses (y) are for the N-th line, a computation is performed using data of that line and the (N-2)-th line and data of the horizontal addresses (x) as apparent from the above. Accordingly, luminance signals can be produced with a horizontal pitch of 1/2Px, ensuring resolution more than twice the normal resolution. As an object in monochromatic mode is expressed only by the signals of the vertical lines, the vertical resolution becomes two times the normal resolution.
When the color filter with the color filter array shown in FIG. 21 is shifted in accordance with the shifting pattern shown in FIG. 24, an equivalent array as shown in FIGS. 25A and 25B is obtained. As this equivalent array is quite the same as that of the color filter shown in FIG. 20, a high definition color picture can be obtained as in the case of the color filter shown in FIG. 20.
In the case of the color filter array shown in FIG. 22, the color CCD 1 is shifted vertically by Py and obliquely by (1/2Px-1/2Py) and (1/2Px+1/2Py) in accordance with the shifting pattern whose four positions (1) to (4) correspond to the four vertexes of parallelogram respectively, as shown in FIG. 27. An equivalent array acquired by this shifting is illustrated in FIGS. 28A and 28B. In this array, the color filter elements F1 are arranged in a checkered pattern with a horizontal pixel pitch of 1/2Px and a vertical pixel pitch of 1/2Py, and the filter elements F2 and F3 are arranged vertically in every line at a pixel pitch of 1/2 and alternately every third column in the horizontal direction. In other words, this array is the same as the array shown in FIGS. 25A and 25B flipped sideways. Therefore, if the signal processing is performed with the horizontal addresses and the vertical addresses exchanged with each other in the computation, a high definition color picture can be obtained with the same matrix as shown in FIGS. 25A and 25B. The shifting pattern shown in FIG. 24 can be applied to the color filter element array shown in FIG. 21.
Although the array of the color filter elements W, Ye and Cy has been discussed in the foregoing description, the array is not limited to this particular type, but this invention can be applied to color filters with the following arrays as arrays including a complementary color filters.
This invention may be applied to a color filter as shown in FIG. 29 (F1: G (green), F2: R (red), and F3: Cy (cyan)), a color filter as shown in FIG. 30 (F1: G (green), F2: Ye (yellow), and F3: Cy (cyan)), and a color filter as shown in FIG. 31 (F1: Y (luminance), F2: R (red), and F3: B (blue)).
The Y (luminance) indicates filter elements having the spectral transmission characteristic of luminance.
Even with the above arrays containing a complementary color filter, a high definition color picture can be acquired by optimizing the shifting pattern.
A description will now be given of a high definition system as another embodiment, which uses four color filter elements shown in FIG. 32 (Ye (yellow), Cy (cyan), Mg (magenta) and G (green)). First, the luminance signal (Y), and color difference signals (R-Y, B-Y) in normal mode are produced by the following computation, for example.
The signals for the pixels Mg 2(y),3(x) are produced from the following equations.
The signals for the pixels Ye.sub.3(y),3(X) are produced from the following equations.
S.sub.YeY,X, S.sub.CyY,X, S.sub.MgY,X and S.sub.GY,X express output signals corresponding the horizontal addresses (X) and vertical addresses (Y), and with S.sub.Ye =R+G, S.sub.Cy =G+B and S.sub.Mg =R+B, R, G and B are normalized to "1." .alpha.1, .alpha.2, .beta.1, and .beta.2 are matrix coefficients.
A description will now be given of an HD recording mode.
FIG. 33 illustrates the shifting of the color CCD 1 in HD recording mode. First, the HD mode is selected by the mode selector 10, and four frames of sync signal pulses are output from the sync signal generator 3. Picture signals are read from the color CCD 1 first in the normal state (original position (1)), and are written in the buffer memory 6. When one frame of picture signals are written in the buffer memory 6, the color CCD 1 is shifted horizontally from the original position (1) to the position (2) by Px, to the position (3) (-1/2Px+1/2Py) and to the position (4) (1/2Px+1/2Py) and picture signals are read at those positions. The read picture signals are then written in the buffer memory 6. The positions (1) to (4) correspond to the four vertexes of parallelogram, respectively.
The picture signals acquired at the four positions in this manner equivalently become signal acquired from the pixels arranged as shown in FIG. 34. In FIG. 34, Fa corresponds to the Ye (yellow) and Cy (cyan) filter elements, and Fb corresponds to the G (green) and Mg (magenta) filter elements. It is apparent from the drawing that all the color filter elements are arranged horizontally by a pixel pitch of 1/2Px, and the filter elements W are arranged vertically by a pitch of 1/2Py while the lines of thee Ye and Cy filter elements and the lines of the G and Mg filter elements are alternately arranged, two lines each, in the vertical direction. As far as the R, G and B components are concerned, all the components are included for both Fa and Fb, so that all the components are arranged horizontally and vertically by a pixel pitch of 1/2. Accordingly, high definition can be provided.
An example of the signal processing for the above array will be described below.
For the vertical addresses with y numbered 1, 3, 5, . . . , processing is performed with respect to the horizontal addresses with x numbered 1, 3, 5, . . . , and for the vertical addresses with y numbered 2, 4, 6, . . . , processing is performed with respect to the horizontal addresses with x numbered 2, 4, 6, and so on.
The computation for the pixels Fa 5(y),3(x) is performed as follows.
The computation for the pixels Fb4(y) 4(x) is performed as follows.
In the production of the luminance signal, as described above, the luminance signal can be produced line by line and at a pitch of 1/2Px, so that the horizontal and vertical resolutions are improved to about two times the normal resolutions. It should however be understood from the equations that for the vertical low-frequency components, if (S.sub.cy +S.sub.Ye) is not equal to (2S.sub.G +S.sub.Mg), line crawl of the luminance signal may occur, which requires correction. This correction is performed as follows. If the white balance of the signals (S.sub.Ye, S.sub.Cy, 2.sub.SG and S.sub.Mg) read from the image sensor is obtained, for a monochromatic object, the equations (39) and (42) for the luminance signal equal each other, thus causing no line crawl. The line crawl of the luminance signal may occur when a color object is imaged, i.e., when R-Y.noteq.0 or B-Y.noteq.0. Therefore the occurrence of the line crawl can be suppressed by providing a correction table determining the amount of correction between lines of the luminance signal in accordance with the levels of the color difference signals (R-Y, B-Y) and performing the correction on the luminance signal with that amount.
A description will now be given of an embodiment in the case where the color CCD 1 is shifted in accordance with a shifting pattern wherein positions (1) to (4) correspond to the four vertexes of parallelogram, respectively, as shown in FIG. 35.
The color CCD 1 is shifted horizontally to the position (2) by Px, and obliquely to the position (3) by -1/2Px+Py and to the position (4) by 1/2Px+Py in the same manner as done in the previous embodiment. FIG. 36 illustrates an equivalent arrangement of the color filter elements in that case. It is apparent that the Cy and Ye pixels and the G and Mg pixels are arranged in a checkered pattern with a horizontal pixel pitch of 1/2 and a vertical pixel pitch of 1/2. The signal processing in this case, i.e., the generation of the high luminance signal Y.sub.H and the color difference signals R-Y and B-Y are performed as follows, for example. The computation for the pixels Fa.sub.5(y),3(x) is carried out as follows.
The computation for the pixels FbS(y),4(x) is carried out as follows.
From the above equations, the horizontal high luminance signal becomes Y.sub.H and the luminance signal can be produced at a pitch of 1/2Px, the acquired resolution is about twice the normal resolution. The vertical low luminance signal is expressed by the equation of Y.sub.L and is produced line by line. As the equivalent array has a pitch of Py as shown in FIG. 36, however, no improvement as high as that on the horizontal resolution is made. But, as no line crawl occurs, the luminance signal between lines need not be corrected.
Even in one-chip complementary color system, high definition is possible by optimizing the shifting pattern.
Although the array of the color filter elements shown in FIG. 32 has been discussed in the foregoing description of this embodiment the array is not limited to this particular type, but the same equivalent array as obtained by the shifting in the previous embodiment is also obtained for the arrays shown in FIGS. 37 to 39.
High definition is provided by writing data obtained at four positions in the previous embodiments. An embodiment of a system for further improving high definition will now be described below.
In the method of the previous embodiments, as in the case of the equivalent array shown in FIGS. 25A and 25B, for example, the color filter elements F1 which assign two pixels for four pixels can be arranged in a checkered pattern with horizontal and vertical pitches of 1/2, but the color filter elements F2 and F3 which assign one pixel for four pixels are vertically arranged at every third line (line after shifting, thus generating lines of insensible pixels. To overcome this problem, the pattern swing as shown in FIG. 40 is performed in this embodiment. In FIG. 40, the reading of picture signals at the positions (1), (2), (3) and (4) is the same as the signal reading at the four positions in the previous embodiments. In this case, swing to four positions can be accomplished with two shifting axes; the shifting position (2) only by Px in the horizontal direction and the shifting position (3) only by -1/2Px+1/2Py in the oblique direction. The position (4) can be obtained by the combination of those two axes. In this embodiment, another axis is added to the two axes. In other words, the shifting axis for the shifting position only by Px in the vertical direction, i.e., to the position (5), is added. The addition of this axis can provided an additional shifting pattern for shifting horizontally from a position (5) to a position (6) by Py+(px), and obliquely to a position (7) Py+(-1/2Px+1/2Py) and (8) Py+(1/2Px+1/2Py). The positions (5) to (8) correspond respectively to the four vertexes of parallelogram, similarly to the positions (1) to (4).
FIG. 42 illustrates an equivalent array that can be obtained at the eight positions (1) to (8). The color filter elements F in this equivalent array represent all the color filter elements F1, F2 and F3. With regard to the color filter elements F1, although overlapping pixels are generated, the overlapping data may be or may not be read. As apparent from the drawing as the three colors (F1, F2 and F3) are all arranged in a checkered pattern with horizontal and vertical pitches of 1/2, the computation for generating color can be accomplished pixel by pixel, and the processing is simple. This system can thus provide a high definition color picture with less occurrence of a false signal than the shifting to four positions.
Although the equivalent array shown in FIG. 42 is the same as the one obtained when the color filter elements in the arrays shown in FIGS. 20 and 21 are shifted in accordance with the shifting pattern shown in FIG. 40 the array is not limited to this particular type. With the array shown in FIG. 42 (F1 is G (green), F2 is R (red) and F3 is B (blue)), this embodiment can be applied to any array with two rows and four columns as a basic unit, like an array with a G stripe-R/G full checkered pattern which is considered best in an electronic still camera to provide the highest picture quality.
Likewise, if the filter elements in the array shown in FIG. 22 (FIG. 21) are shifted at eight positions as shown in FIG. 41, an equivalent array as shown in FIG. 42 can be obtained. In this case this embodiment is not limited to this particular array, but can be applied to any array with four rows and two columns taken as a basic unit as shown in FIG. 44. Further this embodiment is not limited to the color filter array consisting of three colors, but may be applied to a mosaic pattern of Ye, Cy, Mg and G as shown in FIG. 32.
In short, the shifting at eight positions can accomplish higher definition if the restriction on the time of reading picture data is small.
An embodiment shown in FIG. 45 is an electronic still camera which can provide a high-quality picture of a monochromatic object, such as characters, at high resolution without causing color moire. According to this embodiment, a color solid state image sensor (CCD) 31 is connected to a driver 32, and the output terminal of the CCD 31 is selectively connected to a signal processor 36A or a signal processor 36B via a pre-processing circuit 34 and an A/D converter 35. The output terminals of the signal processors 36A and 36B are connected to a recording medium 37, such as a semiconductor memory. Crystal optical filters 38A and 38B are driven by the driver 32, and are selectively arranged in front of the CCD 31. A sync signal generator (SG) 3 receives a mode select signal from a mode selector 39, and supplies sync signals to the preprocessing circuit 34 and A/D converter 35.
To begin with, the normal processing of the electronic still camera shown in FIG. 45 will be described. When the normal imaging mode is selected by the mode selector 39 the crystal optical filter 38A for reducing color moire originated from the color filter array is set in front of the color CCD 31. The signals read from the color CCD 31 are subjected to a predetermined process, such as pre-amplification, white balance or gamma correction, by the pre-processing circuit 34, and are then converted into digital signals by the A/D converter 35. The digital signals are then supplied to the signal processor 36A. This signal processor 36A producers a luminance signal from the picture signal, performs data compression and sends the compressed data to the recording medium 37, such as a memory card.
Then, the monochromatic imaging mode will be described. When the monochromatic imaging mode is selected by the mode selector 39, the crystal optical filter 38B, which suppresses aliasing noise originated from pixel sampling when the color solid state image sensor 31 is considered as a monochromatic image sensor is set in front of the color CCD 31.
The signals read from the color CCD 31 are subjected to a predetermined process by the preprocessing circuit 34, and are then converted into digital signals by the A/D converter 35. The digital signals are then supplied to the signal processor 36B where binarization is performed. Suppose that the color filter array of the color CCD 31 is an array E shown in FIG. 46. FIG. 47 illustrates an equivalent array when the color CCD 31 is used in monochromatic imaging mode. Here, W.sub.G, W.sub.R and W.sub.B means that the color filters R, G and B are white pixels. If the object is monochromatic and white balance is obtained in the pre-processing circuit 34, the signal levels of W.sub.G, W.sub.R and W.sub.B are equal to one another before A/D conversion, so that the threshold levels in the binarization in the signal processor 36B should have the same value for R, G and B.
If white balance for R, G and B is not obtained or when binarization is performed without obtaining white balance, the values for R G and B are checked and the threshold levels in the binarization are separately determined.
A description will now be given of another embodiment association with FIG. 45, referring to FIG. 48. In this embodiment, imaged signals output from a first imaging module, which comprises an optical lens 40, a crystal optical filter 41 and a color solid state image sensor 42, are subjected to a predetermined process, such as preamplification, white balance or gamma correction, by the pre-processing circuit 34, and are then converted into digital signals by the A/D converter 35. The digital signals are then supplied to the signal processor 36A. This signal processor 36A produces a luminance signal and color difference signals from the picture signal, performs data compression and records the compressed data on the recording medium 37, such as a semiconductor memory.
The first imaging module is attachable to and detachable from the electronic still camera, and can be exchanged with a second imaging module. The second imaging module is used, for instance when a picture can be black and white and the resolution is important. The second imaging module comprises an optical lens 43, a crystal optical filter 44 and a monochromatic solid state image sensor 45. The picture signals output from the second imaging module are subjected to a predetermined process by the pre-processing circuit 34 and are then converted into digital signals by the A/D converter 35. The digital signals are then supplied to the signal processor 36A. This signal processor 36A performs monochromatic processing on the received signals and records the resultant signals on the recording medium 37. If the number of pixels of the color solid state image sensor 42 of the first imaging module is set equal to the number of pixels of the monochromatic solid state image sensor 45 of the second imaging module, the same driver 32 and sync signal generator 33 may be used.
As the imaging modules are designed detachable from the electronic still camera as desired, imaging suitable to the purpose becomes possible. If the driver 32 and signal generator 33 are included in the imaging modules, those imaging modules can be exchanged with the imaging module of a solid state image sensor having a different number of pixels (e.g., an HD solid state image sensor).
Referring now to FIG. 49, a description will now be given of how to use an electronic still camera system which can reproduce a picture with a wider image output range than a standard TV monitor, and the concept of the system.
FIG. 49 illustrates a digital electronic still camera 50 having a reproducing function to allow the use of a memory card 51 as a recording medium and the reproduction of the image object on an ordinary TV monitor 54. Although the image output range of the conventional electronic still camera system is fixed, the broken line portion is the image output range of the reproduced image according to the system of this invention. An operator can change the image output range in the horizontal direction within this allowable image output range by operating a playback button 52 and direction buttons 53. Hereinafter, this image outputting method will be called "panorama mode."
A description will now be given of the imaging method which can ensure panorama reproduction. By reading signals at the position corresponding to a 1/2 pitch of the pixels in the vertical direction, using the method shown in FIGS. 1 and 2, the number of pixels in the vertical direction appears to be twice the number of pixels of the ordinary camera.
FIG. 50 illustrates an example of an imaged picture in this case. Provided that the number of the effective vertical pixels of the TV monitor 54 is Nv, the image in FIG. 50 has 2Nv vertical pixels. A half screen in an arbitrary vertical direction is taken as an effective image and is subjected to signal processing accordingly before being recorded on the recording medium, e.g., the memory card 51. In the example of FIG. 50, the unshaded portion is selected. The above manipulation can provide a recorded image having a range in which the image output range can be widened horizontally as shown in FIG. 49.
The reproducing means will now be described. FIG. 51 is a block diagram of a reproducing section 70 of the electronic camera 50. Still picture data recorded on the memory card 51 is sent to a still picture reproducing section of the electronic camera 50 via an interface (IF) 71 between the card and the camera. The data is recorded in an display memory circuit 73 via a digital reproduction signal processor 72. The output data of the display memory circuit 73 is converted into an analog signal by a D/A converter 74. The analog signal is output via an analog signal processing section 75 to the monitor 54.
Provided that the output rate at the time of outputting an ordinary monitor image is fs and the number of the horizontal image pixels then is Nh the effective horizontal image necessary for the panorama mode becomes twice Nh. As the number of the horizontal input pixels in panorama mode is Nh, the correct output image cannot be obtained if the horizontal pixels are output at the rate fs. To ensure the correct image output with the correct aspect the image should be output and reproduced at the rate of fs/2. Alternatively, as shown in FIG. 53 showing the display memory circuit 73, a horizontal interpolation circuit 79 may be provided at the subsequent stage of an display memory 78 so that the image is output at the rate of fs after the effective data is interpolated. FIG. 57 shows an example of the address space of the display memory 78 in this case; the display memory 78 needs a capacity of horizontal Nh.times.vertical Nv pixels. The shaded portion in FIG. 57 shows an example of the memory portion which is selected and its image is output.
In this example, the image portion in nearly the center of the allowable image output range is selected, and an operator can select the left or right direction button 53 shown in FIG. 49 to move the image output range. The selection of the image output direction by the operator is transmitted to a horizontal address counter 76 which generates the addresses of the display memory circuit 73, so that the start address of the horizontal addresses of the display memory 78 is changed, thereby horizontally shifting the image output range.
FIG. 54 shows an example in which the horizontal interpolation circuit is placed at the previous stage of the display memory 78. The memory capacity of the display memory 78 needs to be twice the normal capacity in the horizontal direction as in the case of FIG. 58.
The embodiment with the vertical pixels increased to double has been discussed in the above.
A description will now be given of the case where like the number of the vertical pixels, the number of the horizontal pixels is also increased. This can be realized by HD mode. In the example of FIG. 50, the vertical pixels of 2Nv and horizontal pixels of Nh are input, but in this embodiment the vertical pixels of 2Nv and the horizontal pixels of 2Nh are input. Of those pixels, a half of the vertical pixels are selected and are subjected to signal processing before being recorded on the recording medium. The image data on the recording medium is reproduced by the reproducing section having the display memory circuit 73 in FIG. 55. The address space of the display memory 78 in this case is the same as shown in FIG. 58.
FIG. 59 illustrates an example of another address space which can reduce the capacity of the display memory 78. The display memory 78 has memory space of the vertical pixels of Nv and the horizontal pixels of Nh+Na where 0.ltoreq.Na<Nh. A memory for the horizontal pixels of Na in addition to the horizontal pixels of Nh for the selectively output image is provided, and image data in the designated direction for the Na pixels is put through the signal processor and is supplementarily recorded on the recording medium. Then, the start position of the horizontal addresses is shifted to move the display image. Accordingly, the horizontal address counter should be so designed as to select the head address next to the horizontal final address to provide continuous addresses. FIG. 56 presents a block diagram of the display memory circuit 73 in this case.
The capacity of the display memory 78 can be reduced by supplementing reproduction image data in the above manner as needed. If a block coding scheme such as still picture cosine transform is used in the signal processing, the memory can be used more efficiently by setting Na in accordance with the length of the horizontal block or a multiplication of that length.
The foregoing description has discussed the embodiment which shifts the position of an object image on the image sensor and performs signal reading plural times. A description will now be given of another embodiment which does not shift the position of the formed image.
FIG. 60 shows an example o fusing an image sensor having a wider horizontal image angle. The pixels used in normal imaging mode are the shaded portion in the center and the whole pixels are used in panorama mode The number of horizontal pixels Nb is not particularly specified as long as Nb>Nh.
FIG. 61 shows an image with a mark put in a panorama reproduction image to distinguish the reproduced image picked up in normal imaging mode from the reproduced image picked up in panorama mode. "P" at the upper right of the reproduced image represents a panorama image. At the time of reading data from the memory card, the reproducing section of the electronic camera reads image control data from the memory card which indicates whether the image data is an ordinary image or a panorama image, and selects signal processing matching the image. As one process in this processing, when the output image is a panorama image, a mark representing such is indicated in the reproduced image.
FIG. 62 indicates the direction in which an image can be displayed; in this case, it is indicated that display in both directions is possible. In FIG. 63, the direction mark on the image indicates that there is no image in the right direction and the image can be moved only in the left direction. The panorama mark or direction mark may be generated from a mark generator 81 constituted of a digital circuit and may be mixed with the image by a digital mixer 82 as shown in FIG. 64, or may be generated from a mark generator 83 constituted of an analog circuit and may be mixed with the image by an analog mixer 84 as shown in FIG. 65.
FIG. 66 shows an example in which a display section for indicating the displayable direction to the operator is provided at the operation section of an electronic camera. In this example, light-emitting portions are provided in direction buttons 53a and 53b and the light emission of those portions indicates the displayable direction to the operator. In this drawing, only the left direction is lit, indicating that only the leftward movement is possible.
To inform the user of whether the mode is the normal imaging mode or the panorama mode at the imaging time, the mode may be displayed on the liquid crystal display panel on the camera body or may be displayed in the view finder as shown in FIG. 67. The mark "P" at the upper right within the view finder in FIG. 67 indicates the panorama mode. "P" may be positioned in the finder image display. The entire finder is an effective image in normal imaging mode while the center portion excluding the upper and lower shaded portions is an effective image range in panorama mode. Means for distinguishing the shaded portions from the center portion may be a shield cover for the real image finder. For an electronic finder, the shaded portions can be distinguished from the center portion by increasing or decreasing the level of the luminance signal or increasing or decreasing the level of part of the color signals, e.g., any of the RGB signals, or the levels of two types of signals. FIG. 68 presents the display in the finder when an image sensor with a wide horizontal image angle shown in FIG. 60 is used; the center portion is an effective image range in normal imaging mode while the entire portion is an effective image range in panorama mode.
Although the panorama function in the horizontal direction has been discussed in the foregoing description, the display image may be expanded in the vertical direction or both in vertical and horizontal directions. For the expansion in the vertical and horizontal directions, the entire portion in FIG. 50 should be treated as an effective image range whereas the half portion is the vertical effective image range FIG. 69 is a block diagram of the display memory circuit 73 according to an embodiment in which an image sensor having the vertical pixels of Nv and the horizontal pixels of Nh are shifted twice in the vertical and horizontal directions to change the image forming positions, thereby seemingly increasing the number of pixels to four times. FIG. 70 shows an example of the display image. As shown in FIG. 69 direction information is input to a vertical address counter 77 as per the horizontal address counter 76 In the example of FIG. 70, it is indicated that the image can be moved in the up and down directions as well as in the right and left directions. In this case, an image sensor having a longer horizontal length and a larger number of pixels than the image sensor for the normal TV should be used.
This image sensor may be a Hi-vision image sensor having about twice the number of the vertical pixels.
An embodiment shown in FIG. 71 is an electronic still camera which can add position information indicating the imaging location to the imaging condition data. This electronic still camera comprises a solid state image sensor (CCD) 91, an image signal processor 92, a digital picture signal 93, an imaging condition data recording circuit 94, a memory card interface 95 and a memory card 96.
In FIG. 71, position information is added as imaging condition data. Position data indicating the location of imaging is generated in the camera or such position data is externally input, and this position data is recorded via the imaging condition data recording circuit 94 together with image data on the recording medium. The embodiment of FIG. 71 is a digital electronic still camera which record digital picture signals. The memory card 96 is used as a recording medium. The following description will also be given with reference to a digital electronic still camera.
As means for generating position data position data from a global positioning system (GPS) may be used, position data of a mobile telephone station may be used, discrimination may be made using the radio broadcasting such as FM broadcasting or AM broadcasting, which can be received at the location of the imaging, TV broadcasting, or an input through a keyboard or pen input board may be used.
FIG. 72 illustrates a GPS device connected to an electronic camera to input position data to the camera. At the same time as the imaging is performed or after the imaging, position data is read from the GPS device and is recorded, linked with the image data on the memory card. FIG. 73 is a block diagram of the reproducing section of an electronic camera or the reproducing section of an electronic album having a medium that stores and holds a large amount of image data picked up by an electronic camera. The reproducing section comprises an image data storage section 101, an image data storage interface 102, a reproduction signal processor 103, an image display section 104, a CPU 105, an input section 106, a retrieval question display section 107 and a retrieval data storage section 108.
The large capacity medium may be an optical disk, magneto optical disk hard disk, magnetic tape, magnetic bubble memory, etc. A user who wants to retrieve an image inputs the name of the imaging location through input means in the reproducing section. Then, the position data of that location is read from the retrieval data storage section, that position data is compared with the position data of each recorded image and the image data with a small error is output as being likely one. This image outputting ensures highly efficient image retrieval.
FIG. 74 illustrates a mobile telephone connected to an electronic still camera to input position data to the camera. The electronic still camera is connected to the nearby relay station via the mobile telephone, and identification data of that relation station is input to the camera. As the position information of the identification data station is given to the reproducing section, the approximate location of the imaging is identified. The location of the imaging can be identified more specifically by utilizing the identification data of a plurality of relay stations.
In the examples of FIGS. 72 and 74, position data is input to the electronic still cameras from an external system, but the cameras may be provided with this function of acquiring position data.
FIG. 75 shows an example in which radio waves or TV waves or both are used as means for inputting position data. The tuning frequency of an internal or external tuner 98 is selected from a plurality of frequencies previously set in a frequency data memory 97, and when a tuned output is generated, its output frequency is turned into data which is then recorded on the memory card 96 via the imaging condition data circuit 94.
As the frequency data of the broadcasting stations and relay stations are recorded in the retrieval data storage section in the reproducing section, the location of the image can be estimated by comparing this frequency data with the tuned frequency data in the memory card 96.
The present invention can be used together with the method of using various imaging condition data that has been described in Jpn. Pat. Appln. No. 2-234492, entitled "Digital Electronic Still Camera System." FIG. 76 shows an example of the retrieval screen in the reproducing section, in which position data is used together with other imaging condition data. When a retriever answers the questions on the screen, the reproducing section computes the probable values of each image based on the answers, compares those values with one another, and output the image with the most likely value first. FIG. 77 shows the state of the CPU 105 which performs computation. In accordance with the questions, imaging condition data of each image necessary for the computation is read out and is substituted into predetermined equations. The results of the individual computations are collected by a total computing section to be probability values of the individual images. The values are used in the comparison.
Although digital systems have been described in the foregoing description of the embodiments, this invention can also be applied to an analog system.
An embodiment shown in FIG. 78 is an electronic still camera system equipped with a zoom function and a compressing function. FIG. 79 presents a flowchart illustrating the control flow of this system. In FIG. 78, an image pick-up section 201 includes an optical system having a lens, shutter, etc., and a photoelectric converting element, such as a CCD, for photoelectrically converting an image formed by the optical system. The picture signals output from the image pick-up section 201 are input to an analog signal processor 202 where the signals are subjected to gamma correction or white balance adjustment. The output of the signal processor 202 is converted into digital data by an A/D converter 203, and this digital data is input as digital image information to a digital signal processor 204 including an interpolation processor.
When the zoom mode is selected by a user at the imaging time, the video image is electronically enlarged through linear interpolation or the like in the interpolation processor. The enlarged image data is input to an image compression circuit 205. The compression system may be a color still picture coding system which is one type of block coding such as JPEG. The video image compressed by this system is recorded in a memory pack, e.g., a video image recording medium 206, such as an IC card, which is detachably or exchangeably installed in or connected to the body of the electronic camera.
As described above this electronic zoom system performs image compression after executing interpolation, thereby eliminating the deterioration of the image quality originating from an increase in block deformation caused by the block coding, and suppresses the high-frequency component of the image through the interpolation, thereby suppressing the occurrence of block deformation due to image compression.
A description will now be given of the JPEG as the block data compression system referring to FIGS. 80, and 81A through 81F. According to the JPEG, the original image shown in FIG. 81A is separated into 8.times.8 pixel blocks as shown in FIG. 81B, and discrete cosine transform (DCT) is performed in that blocks. The high-frequency component of a DCT coefficient is eliminated by using a quantization table for roughly quantizing the high frequency to thereby accomplish highly efficient compression. But, the decoded image has a noncontinuity of the frequency component at the boundary of the blocks as shown in FIGS. 81C and 81D causing block deformation. If interpolation is performed after compression, therefore, the block-coding originated block deformation is enlarged as shown in FIG. 81E or 81F to be sensible to human eyes, making the deterioration of the image quality more prominent.
The interpolation has a low pass characteristic as shown in FIG. 82, the amount of information of a high-frequency region is reduced at the time image compression is performed. As mentioned above, the block data compression system is accomplished by reducing the amount of information of the high-frequency region. Since the amount of information of the high-frequency region is already reduced by the interpolation the image information is not affected by the elimination of the high-frequency component in the compression process, so that the block deformation is suppressed.
The system of the present invention is not easily affected by the block coding and can suppress the deterioration of the image quality.
With regard to zooming, zooming by an optical system and electronic zooming may of course be used together.
FIG. 83 illustrates an embodiment equipped with a function for changing the compression ratio by the zooming magnification and a function for checking a read image. The electronic still camera according to this embodiment has a controller 210 as a user interface so that the user can select the functions of the camera system such as the zooming function and image checking function. The content of the selected process is displayed on a display section 211, allowing the user to confirm the content.
The magnification is selected through the controller 210 by the user. As the magnification is increased, the amount of information of the high-frequency component on the spatial frequency is reduced in the case of electronic zooming. If the compression ratio of the block coding for roughly quantizing the high-frequency component is increased, therefore, its influence on the image quality is not high. The zoom magnification at the imaging time is input to a compression ratio selector 212, and a high compression ratio is selected as the magnification increases. This can save the memory without degrading the image quality.
When the function of checking the image supplied to the controller 210 is selected, image information from the image compressor circuit 205 or the recording medium 206 is sent to a decoder 213, and the decoded image is stored in a frame memory 214. This image information is output on a monitor 215 equipped on this image pick-up device or an external monitor terminal 216. Accordingly, it is possible to check whether or not the picked-up image is good before recording the image on the recording medium, such as an optical disk, magnetic disk or digital tape, or before printing.
Although this embodiment is an electronic camera which deals with digital still pictures, this system is effective when used in compression which involves block-coding originated quality deterioration in a digital motion picture camera.
FIG. 84 is an image pick-up device according to the embodiment which has a function for selecting the proper processing at the time of low magnification or high magnification in zoom mode, and has processes suitable for the selection. FIG. 85 is a flowchart for controlling the image pick-up device in FIG. 84.
When the user selects the zoom mode at the imaging time, compression/segmentation process selecting section 221 determines whether or not to compress data. For high magnification, the process selecting section 221 selects the output of an image segmentation process circuit 222 without performing compression in the compression circuit 205. This process selecting section 221 selects the proper processing from the compression ratio of the compression circuit and the zoom magnification.
As the zoom magnification is increased, the number of real pixels in the region of the zoomed image becomes very small, so that the recording of this region alone needs a small amount of data, thus saving the memory. Further, deterioration of the interpolated image caused by the compression-originated deformation of the image will not occur. This system can save the memory and improve the quality of the zoomed image.
FIG. 87 illustrates the operational principle of the compression/segmentation process selecting section 221. Based on the compression ratio and zoom magnification, the user makes an input to the process selecting section 221. This process selecting section 221 has a function f(x) inside. When y.ltoreq.f(x) where x is the compression ratio and y is the magnification, which indicates that the magnification is smaller than the compression ratio, the process selecting section 221 selects compression. When y>f(x), which indicates that the magnification is higher than the compression ratio the process selecting section 221 selects segmentation. An example of the internal function f(x) is f(x)=.sqroot.x, and this function is previously given to the system. This device can perform the proper process so that the best image under the selected conditions can always be obtained.
An example of the image format is shown in FIG. 86. Additional information, such as the size of the image or imaging date, is included in the header portion, and a flag indicating whether segmentation has been performed and zooming is necessary is provided.
As shown in FIG. 88, a process selecting section 224 of a reproduction process section 223 selects the reproduction process in accordance with this flag. If data is compressed, it is decoded by a decoder 213, and the decoded result is stored in a frame memory 214 before being output to a monitor 215 or the like. If segmention is performed, the aforementioned bi-linear interpolation or high dimensional interpolation is performed on the image data, stored on the recording medium, by an electronic zoom processor 225.
Although the embodiment shown in FIGS. 51 through 57 has been described as being applied to a still picture system, the application of this invention to a motion picture system will now be described.
FIG. 89 is a block diagram showing the schematic structure of a digital camera system according to an embodiment which uses the aforementioned electronic zooming system and digitally records motion pictures. In FIG. 89, the image pick-up device 201 includes a photoelectric converting element, such as a CCD, which photoelectrically converts the image formed by the optical system including a lens. The picture signals output from the image pick-up device 201 are supplied to the signal processor 202 where gamma correction, white balance adjustment, and the like are performed. The output of the signal processor 202 is converted into digital data by the A/D converter 203, this digital data is input as digital image information to the digital signal processor 204 including an interpolation processor. When the zoom mode is selected by the user at the imaging time, video image is electronically enlarged through linear interpolation or the like in the electronic zoom processor 207. The enlarged image data is input to the image compression circuit 205.
The compression system may be an MPEG, which is a motion picture coding system that is one type of block coding. The video image compressed by this system is recorded via a recording signal processor 301 and an amplifier 302 on a digital tape 303 or a video recording medium such as an optical disk, which is detachably or exchangeably installed in or connected to the body of the electronic camera.
As described above, this elecronic zoom system performs image compression after executing interpolation by the electronic zoom processor 207, thereby eliminating the deterioration of the image quality originating from an increase in block-coding originated block deformation, and suppresses the high-frequency component of the image through the interpolation, thereby suppressing the occurrence of block deformation due to image compression.
Further, if a communication interface 309 is provided as shown in FIG. 90, this camera system can be used in a TV conference system and a TV telephone system, which outputs a video image on a TV monitor 314 at a remote site via a codec 310, a communication channel 311, a codec 312, and a communication interface 313.
As described above when this electronic zoom system is used in a motion picture system, the block-coding originated deterioration of the image quality can be suppressed, the compression can save the necessary memory, and the amount of communication can be reduced.
Additional advantages and modifications will readily occur to those skilled in the art. Therefore the invention in its broader aspects is not limited to the specific details, and representative devices shown and described herein. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents.
Claims
What is claimed is:
1. An electronic device comprising: an image sensor for outputting an image signal based upon an optical image that is incident thereon, the image sensor including: a plurality of optical filters arranged in both a horizontal and a vertical direction; and a plurality of pixels located at positions corresponding to positions of the optical filters; and a moving unit for moving the optical image in one of the horizontal direction and the vertical direction, and for moving the optical image in an oblique direction among first second, third and fourth positions, the first, second, third and fourth positions defining four vertexes of a parallelogram the first position being a current position of the optical image, the second position being shifted by a distance of one pixel pitch from the first position in one of the horizontal and vertical directions, the third position being shifted by a distance of a half pixel pitch from the first position in both the horizontal and vertical directions, the fourth position being shifted a distance from the first position so as to form the fourth vertex along a diagonal of the parallelogram; the image signal outputted by the image sensor being readable at each of the first, second, third and fourth positions.
2. An electronic device comprising: an image sensor for outputting an image signal based upon an optical image that is incident thereon, the image sensor including: a plurality of optical filters arranged in both a horizontal and a vertical direction; and a plurality of pixels located at positions corresponding to positions of the optical filters; and a moving unit for moving the image sensor in one of the horizontal direction and the vertical direction, and for moving the optical image in an oblique direction among first, second, third and fourth positions, the first, second, third and fourth positions defining four vertexes of a parallelogram, the first position being a current position of the image sensor, the second position being shifted by a distance of one pixel pitch from the first position in one of the horizontal and vertical directions, the third position being shifted by a distance of a half pixel pitch from the first position in both the horizontal and vertical directions, the fourth position being shifted by a distance from the first position so as to form the fourth vertex along a diagonal of the parallelogram; the image signal outputted by the image sensor being readable at each of the first, second, third and fourth positions.
3. An electronic device comprising: an image sensor for outputting an image signal based upon an optical image that is incident thereon, the image sensor including: a plurality of optical color filters arranged in both a horizontal and a vertical direction, the optical filters including: mosaic filters, having filter elements, arranged in both the horizontal and vertical directions, each of the mosaic filters being arranged in a unit in which the filter elements are arranged in one of a pattern of two rows and two columns, a pattern of four rows and two columns and a pattern of two rows and four columns, the filter elements being one of three-color and four-color filter elements; and a plurality of pixels located at positions corresponding to positions of the optical color filters; and a moving unit for moving the optical image in one of the horizontal direction and the vertical direction, and for moving the optical image in an oblique direction among first second, third and fourth positions, the first, second, third and fourth positions defining four vertexes of a parallelogram, the first position being a current position of the optical image, the second position being shifted by a distance of one pixel pitch from the first position in one of the horizontal and vertical directions, the third position being shifted by a distance of a half pixel pitch from the first position in both the horizontal and vertical directions, the fourth position being shifted by a distance from the first position so as to form the fourth vertex along a diagonal of the parallelogram; the image signal outputted by the image sensor being readable at each of the first, second, third and fourth positions.
4. An electronic device according to claim 3, further comprising signal processing means for producing a single still image by processing the image signals outputted by the image sensor at the first, second, third and fourth positions.
5. An electronic device according to claim 3, wherein the moving means moves the optical image from the first position to one of the other three positions, then moves the optical image to one of the remaining two positions, and then moves the optical image to the only remaining position.
6. An electronic device according to claim 5, wherein the moving means returns the optical image to the first position after being moved to each of the first, second, third, and fourth positions.
7. An electronic device according to claim 3, wherein the image sensor further includes means for draining electric charges, which have accumulated on the image sensor, when the moving means moves the optical image.
8. An electronic device according to claim 3, wherein the image sensor further includes means for draining accumulated electric charges, when the moving means moves the optical image.
9. An electronic device according to claim 3, wherein the image sensor further includes means for recognizing a signal outputted by the image sensor, which contains accumulated electric charges, when the moving means moves the optical image.
10. An electronic device according to claim 3, wherein the moving means includes means for electrically moving the image sensor.
11. An electronic device according to claim 10, wherein the moving means includes a piezoelectric element.
12. An electronic device according to claim 3, wherein the moving means includes a parallel-plate member, disposed in front of the image sensor, for changing an optical path along which the optical image is incident on the image sensor.
13. An electronic device according to claim 3, wherein the one of three-color and four-color filter elements includes one of a color green and a color that includes a green-color component, and wherein a pixel arrangement of the one of the color green and the color that includes a green-color component, which is obtained after the image signal is outputted by the image sensor in the first, second, third and fourth positions, is formed in a checkered pattern with a half pixel pitch in the horizontal and the vertical directions.
14. An electronic device according to claim 3, wherein the filter elements are divided into a plurality of filter element blocks each filter element block including four filter elements which are arranged so that two of the four filter elements are in the vertical direction and the other two of the four filter elements are in the horizontal direction, and wherein two of the four filter elements each include a green color filter element.
15. An electronic device according to claim 14, wherein the green color filter elements are arranged in the oblique direction.
16. An electronic device according to claim 14, wherein two of the four filter elements include a red color and a blue color filter element.
17. An electronic device according to claim 16, wherein the green color filter elements are arranged in the oblique direction.
18. An electronic device comprising: an image sensor for outputting an image signal based upon an optical image that is incident thereon, the image sensor including: a plurality of optical color filters arranged in both a horizontal and a vertical direction, the optical filters including: mosaic filters, having filter elements, arranged in both the horizontal and vertical directions, each of the mosaic filters being arranged in a unit in which the filter elements are arranged in one of a pattern of two rows and two columns, a pattern of four rows and two columns and a pattern of two rows and four columns, the filter elements being one of three-color and four-color filter elements; and a plurality of pixels located at positions corresponding to positions of the optical color filters; and a moving unit for moving the image sensor in one of the horizontal direction and the vertical direction, and for moving the optical image in an oblique direction among first, second, third and fourth positions, the first, second, third and fourth positions defining four vertexes of a parallelogram, the first position being a current position of the image sensor, the second position being shifted by a distance of one pixel pitch from the first position in one of the horizontal and vertical directions, the third position being shifted by a distance of a half pixel pitch from the first position in both the horizontal and vertical directions, the fourth position being shifted by a distance from the first position so as to form the fourth vertex along a diagonal of the parallelogram; the image signal outputted by the image sensor being readable at each of the first, second, third and fourth positions.
19. An electronic device according to claim 18, further comprising signal processing means for producing a single still image by processing the image signals outputted by the image sensor at the first, second, third and fourth positions.
20. An electronic device according to claim 18, wherein the moving means moves the image sensor from the first position to one of the other three positions, then moves the image sensor to one of the remaining two positions, and then moves the image sensor to the only remaining position.
21. An electronic device according to claim 20, wherein the moving means returns the image sensor to the first position after being moved to each of the first, second, third, and fourth positions.
22. An electronic device according to claim 18, wherein the image sensor further includes means for draining electric charges, which have accumulated on the image sensor, when the moving means moves the image sensor.
23. An electronic device according to claim 18, wherein the image sensor further includes means for draining accumulated electric charges when the moving means moves the image sensor.
24. An electronic device according to claim 18, wherein the image sensor further includes means for recognizing a signal outputted by the image sensor, which contains accumulated electric charges, when the moving means moves the image sensor.
25. An electronic device according to claim 18, wherein the moving means includes means for electrically moving the image sensor.
26. An electronic device according to claim 18, wherein the moving means includes a piezoelectric element.
27. An electronic device according to claim 18, wherein the moving means includes a parallel-plate member, disposed in front of the image sensor, for changing an optical path along which the optical image is incident on the image sensor.
28. An electronic device according to claim 18, wherein the one of three-color and four-color filter elements includes one of a color green and a color that includes a green-color component, and wherein a pixel arrangement of the one of the color green and the color that includes a green-color component, which is obtained after the image signal is outputted by the image sensor in the first, second, third and fourth positions, is formed in a checkered pattern with a half pixel pitch in the horizontal and the vertical directions.
29. An electronic device according to claim 18, wherein the filter elements are divided into a plurality of filter element blocks, each filter element block including four filter elements which are arranged so that two of the four filter elements are in the vertical direction and the other two of the four filter elements are in the horizontal direction, and wherein two of the four filter elements each include a green color filter element.
30. An electronic device according to claim 29, wherein the green color filter elements are arranged in the oblique direction.
31. An electronic device according to claim 29, wherein two of the four filter elements include a red color and a blue color filter element.
32. An electronic device according to claim 31, wherein the green color filter elements are arranged in the oblique direction.
Patent Citations (13)
| Patent | Date | Inventor | Cited By |
|---|---|---|---|
| US3110762 | 1963-11-01 | Frank | |
| US4517603 | 1985-05-01 | Epsztein et al. | |
| US4633317 | 1986-12-01 | Uwira et al. | |
| US4638371 | 1987-01-01 | Milch | |
| US4641038 | 1987-02-01 | Baker | |
| US4920418 | 1990-03-01 | Robinson | |
| US4967264 | 1990-10-01 | Parulski et al. | |
| US4998164 | 1991-03-01 | Endo et al. | |
| JPX63-284978 | 1988-11-01 | ||
| JPX63-284980 | 1988-11-01 | ||
| JPX378388 | 1991-04-01 | ||
| JPX3231589 | 1991-10-01 | ||
| JPX470275 | 1992-03-01 |
Non-Patent Literature (1)
- Hoagland, Kenneth A., `Image-Shift Resolution Enhancement Techniques for CCD Imagers`, SID 82 Digest, 1982.