Apparatus and method of restoring an image

[0001] This application claims priority from Korean Patent Application No. 10-2007-0092238 filed on Sep. 11, 2007, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety.

[0002] 1. Field

[0003] One or more embodiments of the present invention relate to an apparatus and method of restoring an image, and, more particularly, to an apparatus and method of restoring an image that is capable of extracting a specific channel having a high sharpness, and restoring an image using a specific filter.

[0004] 2. Description of the Related Art

[0005] FIG. 1 is a diagram illustrating output of an RGB image according to the related art. As shown in FIG. 1, when light passes through a lens, the distance at which an image is formed on the basis of the lens depends on the refractive index of the lens. Therefore, light that has various wavelengths forms images at various locations. That is, the light is divided into an R channel, a G channel, and a B channel according to wavelengths, and in each channel, an image is formed at a different location. In particular, when the lens and a sensor are fixed at specific locations, a channel of an image, which is formed on the sensor, is changed according to the location of the subject. As a result, in at least one of R, G, and B channels, an image having the highest sharpness is formed on the sensor.

[0006] Accordingly, a technology is needed for generating an output image having a highest sharpness from an input image using the channel having the highest sharpness among R, G, and B channels on the basis of the above-described fundamental concept.

[0007] One or more embodiments of the present invention provide an apparatus and method of restoring an image that is capable of extracting a channel having a highest sharpness from an input image and restoring the extracted channel through a first specific filter.

[0008] One or more embodiments of the present invention provide an apparatus and method of restoring an image that is capable of extracting a high frequency feature from a restored image, calculating a high frequency feature coefficient from the extracted high frequency feature, and restoring a luminance channel to generate an output image.

[0009] Additional aspects and/or advantages will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the invention.

[0010] According to an aspect of the present invention, there is provided a method of restoring an image, including extracting predetermined channels from image data including a plurality of channels and restoring the extracted channels using a first filter.

[0011] According to another aspect of the present invention, there is provided an apparatus for restoring an image, including an image input unit extracting predetermined channels from image data including a plurality of channels and an image-restoring unit restoring the extracted channels using a first specific filter.

[0022] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0023] FIG. 2 is a block diagram illustrating a structure of an image-capturing system according to an embodiment of the invention.

[0024] An image-capturing system 200 according to an embodiment of the invention includes a lens 210, an aperture 220, an imaging element 230, an ADC (Analog-to-Digital Converter) 240, an image-restoring apparatus 250, a memory 260, a driving circuit 270, a CPU (Central Processing Unit) 280, and an operating unit 290.

[0025] Image light that is obtained from a subject is incident on the imaging element 230 through the lens 210 and the aperture 220. The imaging element 230 may be composed of a CCD, a CMOS, or image-capturing units that are known by those skilled in the art. An analog signal of an image, which is formed on an imaging surface of the imaging element 230, is converted into an original digital image signal by the ADC 240. The converted digital image signal is input to the image-restoring apparatus 250 according to the embodiment of the invention. The image-restoring apparatus 250 processes the input image signal and improves the quality of the output image. The image-restoring apparatus 250 will be described in detail with reference to FIG. 3.

[0026] The image that is processed by the image-restoring apparatus 250 is stored in the memory 260. For example, the memory 260 may be implemented by a non-volatile memory element, such as a ROM, a PROM, an EPROM, an EEPROM, and a flash memory, a volatile memory element, such as a RAM, or a storage medium, such as a hard disk or an optical disk.

[0027] A signal that is output from the operating unit 290 is input to the CPU 280 for system control, and a control signal that is output from the CPU 280 is input to the image-restoring apparatus 250 or the memory 260. As a result, recording of the image that is corrected by the image-restoring apparatus 250 is controlled. Further, the CPU 280 applies a control signal to the driving circuit 270 that controls the aperture 220 or an exposure time (shutter speed) in the imaging element 230 or an AGC (Auto Gain Control) (not shown).

[0028] FIG. 3 is a block diagram illustrating a structure of an image-restoring apparatus 250 according to an embodiment of the present invention. Herein, in the present application, the term apparatus should be considered synonymous with the term system, and not limited to a single enclosure or all described elements embodied in single respective enclosures in all embodiments, but rather, depending on embodiment, is open to being embodied together or separately in differing enclosures and/or locations through differing elements, e.g. a respective apparatus/system could be a single processing element or implemented through a distributed network, noting that additional and alternative embodiments are equally available.

[0029] Referring to FIG. 3, the image-restoring apparatus 250 can operate as a component of the image-capturing system 200, as shown in FIG. 2, but the invention is not limited thereto. The image-restoring apparatus 250 may function as an independent apparatus, and operate in a state where it is mounted in a computer, a PDA (Personal Digital Assistant), a PMP (Portable Multimedia Player), a cellular phone, or the like. Specifically, the image-restoring apparatus 250 may include an image input unit 310, a sharpness-determining unit 320, a first filter 330, an image-restoring unit 340, a high-frequency-feature-generating unit 350, a converting unit 360, a luminance-correcting unit 370, and an RGB-converting unit 380.

[0030] The image input unit 310 receives an image that is stored in a predetermined memory. At this time, the image input unit 310 may separate the received image into each of its channels (for example, for RGB). This is possible because in general, an image does not include a single component but three-dimensional components.

[0031] FIG. 5 is a diagram illustrating a state where an input image is separated for each of the R, G, and B channels in accordance with an embodiment of the invention. Referring to FIG. 5, the input image is separated into each of an R channel, a G channel, and a B channel or corresponding color component images according to wavelengths. If the distance between the subject and the lens is short, the sharpness is generally highest in a B channel 515 among the R, G, and B channels. If the distance between the subject and the lens is neither short nor long, a sharpness is generally highest in a G channel 530 among the R, G, and B channels. If the distance between the subject and the lens is long, the sharpness is generally highest in the R channel 545.

[0032] Referring to FIG. 3 back, the sharpness-determining unit 320 receives the R, G, and B channels or corresponding color components that are separated by the image input unit 310, and determines a channel having a highest sharpness in consideration of the distance between the lens and the subject.

[0033] In the present embodiment, using a high pass filter, the sharpness-determining unit 320 detects edge strength (that is, a high frequency feature) for each of the R, G, and B channels from the input image. If the edge strength is detected, each blur degree can be known, which enables a sharpness of each channel to be known. Accordingly, if the edge strength is measured for each of the channels, it is possible to know which channel has the highest sharpness among the channels.

[0034] As described above, the high pass filter passes a signal that has a frequency higher than a cutoff frequency (boundary frequency between a pass band and an attenuation band) without attenuation, but attenuates a signal that has a frequency lower than the cutoff frequency. Accordingly, since the edge of the signal waveforms that has a high frequency feature has a frequency higher than the cutoff frequency, the corresponding signal can pass through the high pass filter. As one example of the high pass filter, a Laplacian filter may be used.

[0035] The first filter 330 receives a channel that has the highest sharpness from the sharpness-determining unit 320. In consideration of the distance between the lens and the subject and the wavelength, the first filter 330 determines a corresponding specific filter.

[0036] In general, the specific filter may use an experimental method and an assumptive method. Examples of the experimental method may include a method using the distance from a specific variable, and a method using a sigma value of a Gaussian blur filter. Examples of the assumptive method may include a method of estimating a point spread function.

[0037] In the present embodiment, the method of estimating a point spread function is used. The method of estimating a point spread function will now be described in brief.

[0038] First, a point spread function that corresponds to an effective value indicating a blur level is estimated. Preferably, the effective value is compared with effective values of images that are photographed at different distances, the distance between a lens matched with the corresponding effective value and the subject (it may include the distance between an image sensor and the subject) is calculated, and a channel, in which a sharpness according to the calculated distance is highest and a point spread function corresponding to the channel are estimated.

[0039] In order to estimate a point spread function (PSF) that corresponds to the effective value, the image-restoring apparatus according to the embodiment of the invention can use the effective values of images that are photographed at different distances. It is preferable to use a graph or a table that shows a relation between the effective values obtained from images repetitively photographed at different distances off-line and the distance between the subject and the image sensor. That is, it is possible to experimentally calculate the distance between the subject matched with the effective value and the image sensor.

[0040] As described above, the image-restoring unit 340 generates a restored image channel from the channel having the highest sharpness using a specific filter.

[0041] In the present embodiment, according to the image-restoring apparatus, the image-restoring unit 340 applies a convolution filter to the point spread function and removes a blur, thereby restoring an image or improving sharpness. In general, a Wiener filter, a Lucy-Richardson filter, or the like is used as a convolution filter that is applied to a given point spread function to remove the blur.

[0042] For example, the Wiener filter H_w is defined by the following Equation 1.

[0043] $\begin{array}{cc}{H}_w\left(u,v\right)=\frac{D^{*}\left(u,v\right)}{D^{*}\left(u,v\right)D\left(u,v\right)+\frac{S_n\left(u,v\right)}{S_i\left(u,v\right)}}& \mathrm{Equation}1\end{array}$

[0044] In this case, u and v denote two coordinate components in a frequency domain, D(u, v) denotes a PSF whose refined PSF is converted into the frequency domain, and D*(u, v) denotes the conjugate of D(u, v). S_n(u, v) denotes a power spectrum of a noise, and S_i(u, v) denotes a power spectrum of an ideal image. If the Wiener filter is applied to a region of an image on the basis of pixels the Wiener filter H_w filters and a convolution between the Wiener filter H_w and the region of the image (total sum of products between the corresponding values) is calculated, and the pixels are filtered. As a result, a pixel value (a deblurred result) is calculated.

[0045] FIG. 6 is a diagram illustrating a difference in sharpness between restored images in accordance with an embodiment of the invention. Referring to FIG. 6, if the distance between the lens and the subject is short, a sharpness is generally highest in the B channel. If the R channel 610 having a low sharpness is restored, a restored image 620 having a low sharpness is shown. In contrast, if the B channel 630 having a high sharpness is restored, a restored image 640 having a high sharpness is shown. In this way, the channel having the highest sharpness is determined and restored or has its sharpness improved.

[0046] The restored image channels and the blur channels (channels among the R, G, and B channels, except for the channel having the highest sharpness) are transmitted to the converting unit 360. However, the restored image channels are transmitted to the high-frequency-feature-generating unit 350 and the converting unit 360.

[0047] The high-frequency-feature-generating unit 350 generates a high frequency feature from the received restored image channels. In this case, the high frequency feature includes an edge value or the like. A Sobel mask method is generally used when the high frequency feature is detected from the restored image.

[0048] FIG. 7 is a diagram illustrating an image that is generated by extracting a high frequency feature from a restored image in accordance with an embodiment of the invention. Referring to FIG. 7, an x-axial direction mask 710 and a y-axial direction mask 720 exist. A mask of 3×3 is applied to pixels of 3×3 on the basis of a specific pixel. Components of the mask of 3×3 and the corresponding pixels of 3×3 are respectively multiplied and are summed up. The calculated sum can be calculated for each of the x-axial direction mask and the y-axial direction mask. It is assumed that the two obtained sums are S_x and S_y. In this case, if any one of the two obtained sums S_x and S_y exceeds a predetermined threshold value, it is determined that a pixel at a central location is located at an edge of an image. For example, if the sum S_x of a certain pixel exceeds the threshold value, the pixel belongs to an edge in an x-direction, and if the sum S_y of a certain pixel exceeds the threshold value, the pixel belongs to an edge in a Y direction. An image 740 of the high frequency feature that is extracted from an original image 730 by using a Sobel filter is shown in FIG. 7.

[0049] A correlation between the high frequency feature value and the high frequency feature coefficient, which is estimated by using the Sobel filter, can be represented, for example, by Equation 2 using accumulated data.

[0050] $\begin{array}{cc}\partial =1-\exp \left(-\frac{{\left(\mathrm{high}\mathrm{frequency}\mathrm{feature}\mathrm{value}\right)}^2}{\beta }\right)& \mathrm{Equation}2\end{array}$

[0051] In Equation 2, the high frequency feature coefficient ∂ has a correlation with the high frequency feature value and a value of a parameter β. When a user arbitrarily changes the value of the parameter β, the high frequency feature coefficient ∂ is changed. The sharpness of the restored luminance channel is determined according to the changed high frequency feature coefficient, which will be described in detail below.

[0052] FIG. 8 is a diagram illustrating a changed high frequency feature of an initial high frequency feature 810 according to a change in a high frequency feature coefficient according to an embodiment of the invention. Referring to FIG. 8, the user can arbitrarily change the value of the parameter β, and when the value of the parameter β is decreased, the value of the high frequency feature coefficient 820 is increased. Accordingly, the high frequency feature value has a large value in the high frequency feature coefficient 820. In contrast, when the user increases the value of the parameter β, the value of the high frequency feature coefficient 840 is decreased. An intermediate value of the high frequency feature coefficient 830 is also shown.

[0053] As described above, the specified high frequency feature coefficient becomes a parameter of the luminance-correcting unit 370, and a specific high frequency feature coefficient having an optimal sharpness is determined by using a database that is accumulated through various experiments.

[0054] Referring to FIG. 3 again, the converting unit 360 separates chrominance channels and luminance channels from blur channels and restored image channels that have been transmitted from the sharpness-determining unit 320. The separated luminance channels are transmitted to the luminance-correcting unit 370 and the chrominance channels are transmitted to the RGB-converting unit 380.

[0055] The luminance-correcting unit 370 can correct the luminance channels transmitted from the converting unit 360 by using the high frequency feature and the high frequency feature coefficient transmitted from the high-frequency-feature-generating unit 350. A method in which the luminance-correcting unit 370 corrects the luminance channels is not limited to the following embodiment.

[0056] In one or more embodiments of the present invention, a luminance channel Y, a restored image C′, and a high frequency feature coefficient ∂ have correlations that are represented below, for example, by Equations 3 and 4, and generate a restored luminance channel Y′.
Y′=∂*C′+(1−∂)*Y  Equation 3
Y′=Y−∂*Y+∂*C′  Equation 4

[0057] In the present embodiments, if the luminance channel is corrected using the high frequency feature coefficient ∂, the existing luminance is maintained and color information is preserved in the low frequency feature. In the high frequency feature, a sharpness can be increased by inputting the restored image information and increasing the luminance.

[0058] The image that is restored using Equations 3 and 4 will be described with reference to FIG. 9. FIG. 9 is a diagram illustrating an input image 910, a restored image 920, and a luminance-corrected image 930 according to an embodiment of the invention. Referring to FIG. 9, it can be understood from experiments that a luminance-corrected image 930 is clearer than an input image 910 and a restored image 920.

[0059] If the luminance image is corrected, the RGB-converting unit 380 changes the chrominance channels and the corrected luminance channels to R, G, and B channels and generates the changed output image.

[0060] FIG. 4 is a flowchart illustrating a method of restoring an image according to an embodiment of the invention.

[0061] Referring to FIG. 4, image data, which is stored in a predetermined memory or photographed by a device, such as a camera, is input S405.

[0062] If the image data is input, the image data is separated into each of its channels S410. The image data includes a plurality of channels, such as R, G, and B channels.

[0063] After the channels are separated, it is determined which channel or the channel is selected that has the highest sharpness among the separated channels S415. In the present embodiment, as described above, in order to determine which channel has the highest sharpness, edge strength is extracted by using a high pass filter and a blur degree is calculated from the extracted edge degree. In this way, it is possible to determine the channel having the highest sharpness.

[0064] After the channel having the highest sharpness is determined, the corresponding channel is restored or has its sharpness improved to modify the separated image by using the first filter S420. In the present embodiment, as described above, the first filter can use a filter that is applied to a point spread function, and if the distance and wavelength of the input image are determined, the point spread function can be determined.

[0065] If the point spread function is determined, a restored channel is formed from the channel having the highest sharpness using the first filter. In the present embodiment, as described above, the restored image is restored by using the first filter where the convolution filter is applied to the point spread function.

[0066] The chrominance channels and the luminance channels are separated from the restored channels and the blur channels S425. If the separated channels are the luminance channels S430, the high frequency feature is generated by using the second specific filter S435, as described above. In the present embodiment, the second specific filter includes a Sobel filter or the like. After the high frequency feature value (that is, edge value) is extracted, the user uses the value of β as a parameter and defines the correlation between the high frequency feature value and the high frequency feature coefficient ∂ using the above-described Equation 2. Accordingly, if the value of the parameter β is adjusted, the high frequency feature coefficient value is changed. A variety of restored image luminance channels are generated according to the high frequency feature coefficient value S440. If the changed high frequency feature coefficient value according to the value of the parameter β is generated, the restored chrominance channel is generated by operating the high frequency coefficient, the restored channel, and the luminance channel S445. As described above. In the present embodiment, Equations 3 and 4 can be used, and the corresponding luminance channels can be corrected. If the corresponding channels are not the luminance channels S430, the process proceeds to S445.

[0067] If the luminance channel is corrected, the chrominance channel and the restored luminance channel are synthesized, and the output image data is generated S450 and can be displayed on a display. The corresponding channels are converted into the R, G, and B channels that are the same channels as the input images, and the output images are generated and further, the output images can be combined. As described above, the output images are shown in FIG. 9. From FIG. 9, the difference of the sharpness between the images can be appreciated.

[0068] In addition to the above described embodiments, embodiments of the present invention can also be implemented through computer readable code/instructions in/on a medium, e.g., a computer readable medium, to control at least one processing element to implement any above described embodiments and display the resultant image on a display. The medium can correspond to any medium/media permitting the storing and/or transmission of the computer readable code.

[0069] The computer readable code can be recorded on a recording medium in a variety of ways, with examples including magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.) and optical recording media (e.g., CD-ROMs, or DVDs). The computer readable code can also be transferred on transmission media such as media carrying or including carrier waves, as well as elements of the Internet, for example. Thus, the medium may be such a defined and measurable structure including or carrying a signal or information, such as a device carrying a bitstream, for example, according to embodiments of the present invention. The media may also be a distributed network, so that the computer readable code is stored/transferred and executed in a distributed fashion. Still further, as only an example, the processing element could include a processor or a computer processor, and processing elements may be distributed and/or included in a single device.

[0070] While aspects of the present invention has been particularly shown and described with reference to differing embodiments thereof, it should be understood that these exemplary embodiments should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as available for other similar features or aspects in the remaining embodiments.

[0071] Thus, although a few embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the claims and their equivalents.