Image processing apparatus

[0001] 1. Field of the Invention

[0002] This invention relates to image processing, particularly to an image processing apparatus which can be used as an “electronic makeup” apparatus providing high quality and natural looking images in a field where sophisticated works are required in both the contents and representation, such as television dramas and movies.

[0003] 2. Description of the Related Art

[0004] It is known that by applying a two-dimensional ε-filter (for example, refer to Harajima and others “ε-Separating Nonlinear Digital Filter and Its Applications” IEICE (The Institute of Electronics, Information and Communication Engineers) 1982, 4,J65-A, No.4, pp297-304) to an image of a human face, neck, hand, etc., a “smooth skin” image with wrinkles and spots removed or decreased can be obtained (refer to Arakawa and others “Color Face Image Processing by Vector ε-Filter˜Removal of Wrinkles˜” March 1998, 1998 General Conference of IEICE D-11-143, PP143-). This will be hereinafter referred to as conversion to “smooth skin.” This is based on the fact that small-amplitude light and dark change is smoothed by “separating and suppressing small-amplitude high-frequency noise components in an image”, which is a function of the two-dimensional ε-filter.

[0005] The ε-filter was originally designed for the purpose of separating and removing the small-amplitude high-frequency noise components superimposed on a signal waveform.

[0006] A low-pass filter (LPF) that is usually used for noise removal not only suppresses noise components, but also degrades edges of signals. Thus if it is applied to an image, the low-pass filter has the disadvantage of blurring the whole image. However, as shown in the relationship between the input and output signals of the ε-filter in FIG. 1, the ε-filter has a characteristic of flattening only small-amplitude level changes in a signal waveform, and has a feature of scarcely impairing sharpness of the whole image because the edges which have steep level changes are preserved.

[0007] Letting an input signal sequence be x (m, n), the output signal y (m, n) of the two-dimensional ε-filter is represented by the following expression (1): $\begin{array}{cc}y\left(m,n\right)=x\left(m,n\right)-\sum _i\sum _j{a}_{i,j}·F\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)& \left(1\right)\end{array}$
where a_i,j is a weight coefficient. If the filter size is (2M+1) X (2N+1), it will satisfy the following expression: $\begin{array}{cc}\sum _{i=-M}^M\sum _{j=-N}^N{a}_{i,j}=1& \left(2\right)\end{array}$

[0008] The function F(x) represented in expression (1) in a nonlinear function shown in a graph of FIG. 2 and F(x)=0 when |x|>ε_o.

[0009] Throughout the specification, the value of ε_o will be referred to as ε value.

[0010] FIG. 3 shows a basic configuration of the two-dimensional ε-filter.

[0011] In FIG. 3, the solid line frame denoted by numeral 1 indicates a calculation section of a small-amplitude high-frequency noise component u (m, n) (the second term on the right side in expression (1)). Output from the calculation section is subtracted from the input signal sequence x (m, n), whereby the output signal sequence y (m, n) with the small-amplitude noise component suppressed is provided.

[0012] By applying ε-filter to an image of a human face, the face image can be converted into a beautiful face having “smooth skin” with wrinkles and spots removed or decreased. The wrinkles and spots are not so-called noise, but involve comparatively small-amplitude light and dark changes in an image. The wrinkles and spots can be made inconspicuous, by smoothing these small-amplitude light and dark changes with the small-amplitude level change suppressing function of the ε-filter.

[0013] Here, only the small-amplitude level changes such as the wrinkles and spots are flattened and steep level changes such as boundaries of pupils, eyelids, eyebrows, etc., are kept, so that sharpness of the whole image is scarcely impaired and conversion to “smooth skin” can be accomplished.

[0014] Conversion to “smooth skin” is intended for skin areas strictly. However, if the whole image is processed uniformly through the ε filter, small-amplitude level changes of peripheral images are also suppressed and as a result, delicate light and dark patterns of hairs, clothes, backgrounds, etc., are also flattened, resulting in an image where original details and texture are impaired.

[0015] This is fatal in applications wherein high quality is demanded for the whole image such as television, movies, etc. However, the ε filter is made to selectively act only on the skin color area of an image, whereby conversion to “smooth skin” can be accomplished without impairing any details or texture of peripheral images. To do this, a technique called “chromakey” (technique of electronically discriminating a specific color area in an image from others and applying image processing such as filtering on y to the area restrictedly), a well-used device in the television technical field, may be adopted.

[0016] As described above, conversion to “smooth skin” through the ε filter has the potential for use as “electronic makeup” in television drama and movie production, etc. However, to make the most of the ε filter in the field where there are various demands for image processing and requirements for high quality of the whole image, the following problems to be resolved exist:

[0017] Conversion to “smooth skin” makes a skin image seem to be smooth, and thus also produces the “rejuvenation” effect as video representation. For an elderly actress to act as a girl such as in a drama of a biography, etc., although there are existing points to be improved (which will be pointed out later), conversion to “smooth skin” is a promising technique to be used as an electronic makeup technique where “rejuvenating skin” is obtained by performing image processing.

[0018] As described above, conversion to “smooth skin” can be realized using the ε filter. On the other hand, for playing an aged person drastically exceeding the actual age such as in a television drama or a movie, an electronic makeup technique for producing an “old” effect such as “wrinkle enhancement” is also demanded. However, hitherto, it has been impossible to realize “wrinkle enhancement” which can also be said to be the opposite effect to conversion to “smooth skin.”

[0019] In conversion to “smooth skin” using the ε filter, the degree (strength) of conversion to “smooth skin” can be easily adjusted simply by changing one parameter (ε_o in FIG. 2) in response to the strength of the wrinkles and spots to be hidden. Although the whole skin becomes smooth as the degree of conversion to “smooth skin” is increased, the grain and texture of the skin are lost and “plastic” texture rather than “human skin” results, leading to an image poor in reality. To avoid this, if conversion to “smooth skin” is weakened, the natural look is recovered, but the wrinkles and spots to be hidden appear; this is a dilemma.

[0020] In other words, a solution to “provide an image where the grain and texture of the skin are preserved while hiding annoying wrinkles and spots” becomes necessary.

[0021] Further, assuming that the above-described “wrinkle enhancement” can be realized, it is desired to provide a “wrinkles enhanced” image with the grain of the skin kept in the original state rather than simple “wrinkle enhancement,” needless to say.

[0022] It is therefore an object of the invention to realize “wrinkle enhancement” that was previously impossible, and to provide an image processing apparatus for performing conversion to “smooth skin” without impairing the grain or texture of skin, and “wrinkle enhancement” with the grain of the skin preserved in its original state.

[0023] Finally, according to the invention, an image processing apparatus is provided for performing conversion to “smooth skin” or “wrinkle enhancement” for an input image. This apparatus is characterized by input image signals which are applied to an input terminal of a ε-filter, and an extended ε-filter adapted to extract selectively or simultaneously, “wrinkle enhanced” signals obtained by adding the output signal of a small amplitude variation component calculation section of the ε-filter to the input image signal, and “smooth skin” converted signals of the input image signals (which is the output of the previously mentioned ε-filter).

[0024] According to this invention, an image processing apparatus is provided, which is characterized by having the ability to extract an input image signal as a “smooth skin” converted signal while preserving the grain and texture of the skin. This is achieved by applying an input image signal to input terminals of two ε-filters having large and small ε values, and adding the output signal of the small-amplitude variation component calculation section of the smaller ε valued ε-filter to the output signal of the ε-filter having the larger ε value.

[0025] According to this invention, an image processing apparatus is provided, which is characterized by having the ability to get an image signal as a “smooth skin” converted signal or a “wrinkle enhanced” signal correspondingly, while preserving the grain and texture of the skin. This is achieved by applying an input image signal to the input terminal of a large ε valued extended ε-filter and the input terminal of a small ε valued ε-filter. When the correction instruction is “smooth skin” conversion, a “smooth skin” converted signal preserving the grain and texture of the skin is obtained by adding the output signal of the small-amplitude variation component calculation section of the ε-filter to the “smooth skin” converted output signal of the extended ε-filter.

[0026] When the correction instruction is “wrinkle enhancement”, a “wrinkle enhanced” signal preserving the grain and texture of the skin by subtracting the output signal of the small-amplitude variation component calculation section of the ε-filter from the “wrinkle enhancement” output signal of the extended ε-filter.

[0027] According to the invention, an image processing apparatus is provided, which is characterized by having the ability to selectively or simultaneously provide

• • [0028] a “smooth skin” converted signal preserving the grain and texture of the skin,
  • [0029] a “wrinkle enhanced” signal keeping the grain and texture of a skin in its original state. This is achieved by a new type digital filter (named “β-filter”) which separates specific variation components within a specified amplitude band When given any two large and small amplitude values that can be set arbitrarily, the β-filter selectively separates only the variation components which have an amplitude value between the two given amplitude values.

[0042] Referring now to the attached drawings, the preferred embodiments of the invention are shown.

[0043] The invention will be discussed in the following order:

• • [0044] (1) Realization of “wrinkle enhancement”
  • [0045] (2) Realization of conversion to “smooth skin” and “wrinkle enhancement” while preserving the grain and texture of the skin.

[0046] The filter according to the invention, which is capable of providing both functions of “smooth skin” and “wrinkle enhancement” conversion, will be hereinafter referred to as extended ε-filter.

[0047] Item (2) above is realized by the following three methods a to c:

• [0048] a. Two ε-filters represented by expression (1) are used in combination to execute conversion to “smooth skin” with a natural look.
• [0049] b. An ε-filter and an extended ε-filter are used in combination in order to execute switch-able conversion to “smooth skin” and “wrinkle enhancement” with a natural look.
• [0050] c. A new digital filter provided by the invention is used to execute conversion to “smooth skin” and “wrinkle enhancement” with a natural look.

[0051] First, realization of “wrinkle enhancement” in (1) will be discussed.

[0052] It is difficult to make a new wrinkle at a point where no wrinkle exists. In this invention, the output of the small-amplitude variation component calculation section, which is a component of the ε-filter, is used to enhance an original wrinkle.

[0053] To execute conventional conversion to “smooth skin”, the small-amplitude variation component calculation section, indicated by numeral 1 in the ε-filter, finds the small-amplitude variation component of an input signal sequence and the component is subtracted from the input signal sequence (see FIG. 3).

[0054] Conversely, as shown in expression (3), the small-amplitude variation component (second term on the right side of expression (3)) is added to the input signal sequence (first term on the right side), whereby the small-amplitude variation component of a wrinkle, etc., can be enhanced (“wrinkle enhancement”). However, the addition (same phase addition) result might exceed the stipulated level range of a digital image (for example, 0 to 255 in 8 bits range), thus clip processing after addition (processing of limiting the result to 0 if the result is 0 or less and limiting the result to 255 if the result is 255 or more) is required. $\begin{array}{cc}y\left(m,n\right)=x\left(m,n\right)-\sum _i\sum _j{a}_{i,j}·F\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)& \left(3\right)\end{array}$

[0055] It is convenient and cost efficient to integrate the apparatus for realizing “wrinkle enhancement” with the existing apparatus for realizing conversion to “smooth skin”, and enable both conversion to “smooth skin” and “wrinkle enhancement.”

[0056] As the integral circuit configuration, the following two configurations are possible:

• [0057] (a) Configuration wherein either “smooth skin” or “wrinkle enhancement” conversion signal is output (which will be hereinafter referred to as “one-output type”), and the small-amplitude variation components are added/subtracted to/from the input image in response in to an external conversion instruction (=“smooth skin”/“wrinkle enhancement”) and
• [0058] (b) configuration wherein the processed results of conversion to “smooth skin” and “wrinkle enhancement” are output simultaneously, which will be hereinafter referred to as “simultaneous output type,” and either can be selected for use as required.
  (both configurations will be referred to as extended ε-filters throughout the specification.)

[0059] FIGS. 4 and 5 show configuration examples of the invented image processing apparatus for executing conversion to “smooth skin” and “wrinkle enhancement” in the forms corresponding to (a) and (b) above. In FIGS. 4 and 5, the portion of the solid line frame indicated by numeral 1 is a calculation section of a small-amplitude variation component U (m, n). A conversion instruction section indicated by numeral 2 in FIG. 4 is a section for instructing the image processing apparatus to perform either conversion to “smooth skin” or “wrinkle enhancement.” Each apparatus comprises a clip circuit (Clip) for suppressing the addition (same phase addition) result of the small-amplitude variation component for “wrinkle enhancement” to the stipulated level range of a digital image (for example, 0 to 255).

[0060] Next, the method of realizing conversion to “smooth skin” while preserving the grain and texture of the skin by using two ε-filters represented by expression (1) in combination (the method in (2) a. described above) will be discussed.

[0061] First, the principle will be discussed.

[0062] A wrinkle, a spot, and grain of the skin are all comparatively small-amplitude level changes (amplitude variation). However, if a wrinkle and a spot at an annoying level are compared with the grain of the skin, generally the amplitude variation of the grain of the skin is comparably minute. So-called image pickup noise is a possible variation element close to the amplitude variation of the grain of the skin, but the image pickup noise may be ignored in high-quality image pick-up conditions of television and movies.

[0063] In other words, there is the following relationship among them:

[0064] Amplitude variation of wrinkles or spots >amplitude variation of grains and textures of skin >>image pickup noise.

[0065] Incidentally, if a small amplitude value εh adequate for removing wrinkles and spots (conversion to “smooth skin”) is processed as the ε value (see FIG. 2) through the ε-filter represented by expression (1), the level change concerning the grain of the skin will, of course, also be suppressed.

[0066] On the other hand, according to the operation principle of the ε-filter, the minute-amplitude variation component u (m, n), provided by setting a minute amplitude value εL (εL<εh) as the ε value (which matches the amplitude variation for the grain of the skin), and processing it through the ε-filter, is a level change related to the grain and texture of the skin. Therefore, in the invention, it is used to revive the grain and texture of the skin which was lost in previous technique.

[0067] In other words, in order to “preserve the grain of the skin with smaller variation width while removing annoying wrinkles and spots (conversion to “smooth skin”),”

• [0068] (a) the original image is converted to “smooth skin” using a small amplitude value εh through ε-filter I (here, the grain component is also suppressed);
• [0069] (b) the level variation component with minute amplitude (grain component) is separated from the original image using the minute amplitude value εL through ε-filter II; and
• [0070] (c) the level variation component with minute amplitude separated in (b) is added to the processing result of the “smooth skin” conversion in (a).

[0071] FIG. 6 shows a configuration example of the invented image processing apparatus for performing the signal processing described above.

[0072] Next, the method in (2) b. described above will be discussed, namely, the method of using extended ε-filter I and ε-filter II in combination in order to realize switch-able conversion to “smooth skin” and “wrinkle enhancement” with a natural look.

[0073] Also in this method, in order to execute conversion to “smooth skin,” the minute-amplitude variation component u (m, n), provided through the ε-filter II, is added to output of the extended ε-filter I as explained in the method (2) a. described above. Thus the grain and texture of the skin which had been lost in previous technique is revived.

[0074] In contrast, in order to execute “wrinkle enhancement,” enhancement of the grain based on the minute-amplitude variation component u (m, n), which is essentially undesirable, is canceled for restoring the original grain level.

[0075] In other words, in order to “enhance significant wrinkles and spots while preserving the grain of the skin in its original state,”

• [0076] (a) wrinkles of an original image are enhanced according to small amplitude value εh, by using extended ε-filter I (here, the grain component is also enhanced);
• [0077] (b) level variation component with minute amplitude (grain component) is separated from the original image, using the minute amplitude value εL through the ε-filter II; and
• [0078] (c) the level variation component with minute amplitude separated in (b) is subtracted from the processing result of “wrinkle enhancement” in (a).

[0079] FIG. 7 shows a configuration example of the invented image processing apparatus, for selectively performing processing between “preserving the grain of the skin, which has smaller variation width, while removing annoying wrinkles and spots” and “enhancing significant wrinkles and spots while preserving the grain of the skin in its original state.”

[0080] In FIG. 7, a minute amplitude component adjustment section, indicated by numeral 3, is a section for performing adjustment processing of a minute amplitude variation component. This section is for executing the algorithm of the minute amplitude variation component u2 (m, n), which is separated through the ε-filter II for output y1 (m, n) of the extended ε-filter I, in response to the correction instruction for conversion to “smooth skin” or “wrinkle enhancement” (addition if the instruction is conversion to “smooth skin” or subtraction if the instruction is “wrinkle enhancement”).

[0081] In this case, the extended ε-filter I is provided for performing conversion to “smooth skin” or “wrinkle enhancement” in accordance with the correction instruction; this topic was previously described with reference to FIG. 6.

[0082] Next, the method in (2) c. described above, will be discussed. This is the method to realize conversion to “smooth skin” and “wrinkle enhancement” with a natural look by using a new digital filter provided by the invention.

[0083] First, the method in (2) a. described above, will be reviewed. This is the method of using two ε-filters in combination, represented by expression (1), in order to perform conversion to “smooth skin” with a natural look. This method is executed according to the configuration previously described with reference to FIG. 6.

[0084] Signal processing performed according to the configuration in FIG. 6 is represented by expression (4). In expression (4), the first term on the right side (enclosed in braces) indicates the function of the ε-filter I (conversion to “smooth skin” based on ε=εh). The second term on the right side following the first term indicates the function of the ε-filter II (separation of the variation component “related to the grain of the skin” of amplitude ε or less). $\begin{array}{cc}y\left(m,n\right)=\left\{x\left(m,n\right)-\sum _i\sum _j{a}_{i,j}·F_{\varepsilon h}\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)\right\}+\sum _i\sum _j{a}_{i,j}·F_{\varepsilon L}\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)& \left(4\right)\end{array}$
Now, if expression 4 is sorted out by focusing attention on: $\sum _i\sum _j{a}_{i,j}$
the following expression (5) is obtained. $\begin{array}{cc}y\left(m,n\right)=x\left(m,n\right)-\sum _i\sum _j{a}_{i,j}·\left\{F_{\varepsilon h}\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)-F_{\varepsilon L}\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)\right\}& \left(5\right)\end{array}$

[0085] The Fε (x) (see FIG. 2) returns the same value for x with |x|<ε, otherwise it returns 0. Thus, FεH (Δx) and FεL (Δx) becomes the following:

• [0086] When |Δx|<εL, Fεh (Δx)=Δx, FεL (Δx)=Δx
• [0087] when εL<|Δx|≦εh, Fεh (Δx)=Δx, FεL (Δx)=0
• [0088] when εL<|Δx|, Fεh (Δx)=0, FεL (Δx)=0
  Therefore, the whole portion represented by {Fεh (Δx)−FεL (Δx)} in expression 5 has a characteristic of outputting the same value only for the variation component Δx contained in the amplitude band with εL<|Δx|<εh, and otherwise outputting 0.

[0089] This characteristic indicates a new nonlinear function defined by two parameters βL and βh as shown in FIG. 8 (φβL, βh (x)).

[0090] Using the new nonlinear function φβL, βh (x), expression (5) can be rewritten as the following expression (6): $\begin{array}{cc}y\left(m,n\right)=x\left(m,n\right)-\sum _i\sum _j{a}_{i,j}·{\varphi }_{\beta L,\beta h}\left(x\left(m,n\right)-x\left(m+i,n+j\right)\right)& \left(6\right)\end{array}$

[0091] In other words, expression (6) indicates the characteristic of a new nonlinear digital filter, which uses a new nonlinear function φβL, βh (x) to separate and suppress only the variation component of a specific amplitude band. This will be hereinafter referred to as “specific amplitude band variation component separation type digital filter” (or β-filter). As seen from the fact that expression (6) is derived from expression (4), the function is the same as the function provided by using two ε-filters in combination shown in FIG. 6. Therefore, one β-filter can accomplish conversion to “smooth skin” while preserving the texture of the skin in one process.

[0092] The β-filter can provide not only the suppression function of the separated specific amplitude band variation component (conversion to “smooth skin”), but also the enhancement function of the variation component (“wrinkle enhancement”). This is similar to how the expanded ε-filter is provided, by attaching the ε-filter output with the configuration of adding the small amplitude variation component separated through the ε-filter to an input image signal. Thus, as shown in FIG. 9, it is possible for one β-filter to cover high-quality conversion to “smooth skin” and “wrinkle enhancement”, while considering the grain of the skin (in the previously described configuration FIG. 7, two filters of extended ε-filter I and ε-filter II are required). Therefore, an “electronic makeup apparatus” for producing natural processing results in a simple configuration can be provided.

[0093] In the basic configuration of the invented image processing apparatus using the specific amplitude band variation component separation type digital filter (or β-filter), the following two configurations are also possible:

• [0094] (a) “One-output type,” configuration wherein one correction output of conversion to “smooth skin,” etc., is adopted. The algorithm of a specific amplitude band variation component for an input signal (subtraction or addition) is changed in response to an external correction instruction (instruction for performing “smooth skin” or “wrinkle enhancement”); and
• [0095] (b) “Continuously parallel output type,” configuration wherein the processing results of conversion to “smooth skin” and “wrinkle enhancement” are output continuously, and either can be selected for use as required.

[0096] This is similar to the configuration example of the extended ε-filter, provided by adding the function of enhancing a small-amplitude variation component to the configuration of the conventional ε-filter.

[0097] FIGS. 10 and 11 show configuration examples of the invented image processing apparatus for executing conversion to “smooth skin” and “wrinkle enhancement” through the β-filter in the forms corresponding to (a) and (b) above. In FIGS. 10 and 11, the portion of the solid line frame indicated by numeral 4 is a specific amplitude band variation component calculation section. Each apparatus (FIGS. 10, 11) also comprises a clip circuit (Clip) for suppressing the addition (common-mode addition) result of the specific amplitude band variation component for “wrinkle enhancement” to the stipulated level range of a digital image (for example, 0 to 255).

[0098] The β-filter provided by the invention will be further discussed.

[0099] The function φβL, βh (x), which is fundamental part of the β-filter, is characterized by the fact that it is nonlinear function acting only on level variations having an amplitude value between the two amplitude values φβL and βh (x) (φβL≦|x|≦βh), and does not involve minute variation amplitude in the proximity of 0. Both functions of the conventional ε-filter, function φβL, βh(x) and nonlinear function F(x) shown in FIG. 2, define the filter requirement in the amplitude area of level variation; the essential functional difference between both functions φβL, βh(x) and F(x) can be compared to the difference between a band-pass filter and a low-pass filter in frequency.

[0100] In other words, the nonlinear function φβL, βh(x) is introduced, whereby producing digital filter capable of selectively separating and suppressing or enhancing only level variations contained in a specific amplitude band of an input signal sequence consisting of level variations of various amplitudes.

[0101] As previously described with reference to related previous documents, the ε-filter is a filter originally designed for the purpose of removing small-amplitude noise. For the purpose of removing noise, the function with the form as shown in FIG. 8 is not considered as the nonlinear function to be used. This is because in noise removal, it is not possible to remove noise in one amplitude band while saving minute amplitude changes.

[0102] Therefore, the digital filter, which is defined by these nonlinear and linear functions and provided by this invention, was produced from the purpose and idea entirely different from those of previous technique. This enables new applications in each field.

[0103] Finally, by combining the use of the P filter and a color area determination circuit, image processing for executing higher-quality conversion to “smooth skin” or “wrinkle enhancement” while considering the grain of the skin and preserving peripheral details and texture is made possible. The configuration demonstrates further effectiveness as an “electronic makeup” apparatus in fields of television and movies, etc.

[0104] FIG. 12 shows a configuration example of the “electronic makeup” apparatus having such a configuration.

[0105] In FIG. 12, a color area determination circuit 5 determines a specific color area specified in parameters in an input image signal. A changeover switch 6 is controlled, based on the output of the color area determination circuit 5, so as to extract the output y (m, n) of the β filter as the output signal only in the determined color area, and output the original image signal for other areas.

[0106] As described above, the invented image processing apparatus uses the small amplitude variation component separation function of the ε-filter and the specific amplitude band variation component separation function of the βε-filter provided by the invention to correct the input image signal sequence as a two-dimensional image. Thereby executing “wrinkle enhancement” and conversion to “smooth skin” and “wrinkle enhancement” with a more natural look. The concept of separating and weakening or enhancing the small amplitude variation component and the specific amplitude band variation component, which is the basic principle of the invention, can be applied to the processing of one-dimensional signals as so-called time-series signals, needless to say.

[0107] According to this invention, the effects of “rejuvenation” and “old age” with a natural look for video representation in television and movies, can be produced in a more natural form by image processing.