Image processing device and method employing motion detection to generate improved quality image from low resolution image

[0002] The present invention relates generally to image processing, and in particular, to an image processing device and method for reducing aliasing distortion due to undersampling of an image. The invention also relates to a technique for generating a high resolution image by increasing the number of pixels of a low resolution image.

[0003] With modern digital imaging techniques, analog video signals are sampled and stored as digital data for subsequent reproduction on a cathode ray tube (CRT) or other display. It is a design goal to display such images with as high a resolution and as little distortion as possible.

[0004] Images displayed on a CRT, such as images derived from television signals, video tape recorders (VTRs) or from digital versatile disc (DVD) players, are generated by repetitive scanning of electron beams in the horizontal direction. As illustrated in FIG. 25., each horizontal scan is performed from left to right, and after each scan, the beams are swept back to the extreme left and re-positioned vertically to commence the next horizontal scan. The fluorescent screen of the CRT is irradiated by three electron beams to illuminate respective blue, green and red phosphors distributed in small units on the screen. The phosphors produce points of light corresponding to the intensity of the electron beams, and the congregation of all the points produces an image. Hence, a displayed image can be considered a congregation of such points, that is, pixels.

[0005] Since a displayed image on a CRT derived from an analog signal is composed of a congregation of light emitting pixels, the image can be envisioned as a digital signal obtained by sampling the original image at the pixel positions. Thus, if the original analog image were sampled at a sufficient sampling interval in both horizontal and vertical directions to generate the same number of points as the number of pixels on a CRT, a collection of image data could be stored digitally. When subsequently reproduced, an image of nearly the same resolution is obtained as in a strictly analog recording/reproduction approach.

[0006] The sampling theorem dictates that an analog signal can be reconstructed completely from a set of uniformly spaced discrete samples thereof in time, provided that the signal is sampled at a rate of at least twice the highest frequency component of the signal. When sampling an original image, if the sampling theorem is not satisfied, aliasing distortion is generated in the displayed image. To correct for aliasing distortion, prefilters have been used to compensate for undersampling in the horizontal (scanning line) direction; however, such prefilters are not typically provided in the vertical direction. As such, aliasing distortion in the vertical direction is a common problem.

[0007] FIG. 26 illustrates aliasing distortion of a displayed image in the vertical direction. Four pixels P₁-P₄of a given column are shown in each of frames N and N+1. Signals S_Nand S_N+1represent the image level variation of the original image in the vertical direction for the given column, where the amplitude level is illustrated horizontally in the figure. Thus, for example, in frame N, the luminance of the image is higher for pixel P₂than for pixel P₁. Now, if the highest spatial frequency component of the original image in the vertical direction has a period of less than twice the horizontal spacing between pixels, then the sampling theorem is not satisfied and aliasing distortion results. Such is the case for both signals S_Nand S_N+1in FIG. 26. For instance, signal S_N′, which is an approximation for the sampled signal of frame N, is markedly different from the original signal S_N. With the aliasing distortion, the high frequency component of the original signal will be lost during reproduction, even if a filter to remove aliasing is employed in the vertical direction. Such aliasing distortion may be a cause of degradation in signal processing such as Y/C separation, noise removal, quality improvement, and so forth.

[0008] It is noted that while undersampling of the image as just discussed will always result in diminished resolution, the effect on the viewer in terms of picture quality depends on how the scene changes from frame to frame. If a frame with aliasing distortion changes significantly from frame to frame, as is the case in FIG. 26, then an unnatural moving image or blurring results from the perspective of the user. If the scene remains still, the aliasing noise is not as great. In any event, since resolution is always degraded by undersampling, and since the signals of standard television broadcasts and the like are intended for only a limited number of horizontal sweeps per frame, there is a need for a practical way to remove aliasing and recapture the original image with improved quality.

[0009] It is, therefore, an object of the invention to provide a way to convert an undersampled image signal into an output image signal with aliasing distortion reduced or eliminated.

[0010] It is another object of the invention to generate a high resolution image from a low resolution image.

[0011] It is yet another object of the invention to improve the quality of a displayable image generated from a low quality input image signal.

[0012] Various other objects and advantages of the invention will become readily apparent to those of ordinary skill in the art, and the novel features will be particularly pointed out in the appended claims.

[0013] In an illustrative embodiment of the invention, there is provided an image signal converting apparatus for converting a first image signal into a second image signal, where the first and second image signals each include a plurality of pictures of different frames. The apparatus includes a motion detector operable to detect motion of the first image signal between a first frame and a second frame, and processing circuitry for producing the second image signal based on an assumption of a pixel at a position corresponding to the detected motion.

[0014] The apparatus may be employed to produce an output signal of either the same resolution as the first image signal with aliasing distortion reduced or eliminated, or with higher resolution in vertical and/or horizontal directions. The motion detector preferably detects the motion of the first image signal by a finer amount than a pixel size of the first image signal. The processing circuitry may be composed of a resolution creating memory for storing a picture of the first image signal and having a larger storage capacity than an amount of data within one picture of the first image signal, and a controller operable to control writing of the first image signal into the resolution creating memory and reading of a new image signal from the memory. The controller writes the first image signal in the memory in accordance with the detected motion of the first image signal.

[0015] In another embodiment, the image converting apparatus includes an area divider for defining at least first and second image areas of a picture, where motion is detected in each of the image areas. In this case, first and second image areas of the second image signal are generated based on an assumption of pixels at positions corresponding to the detected motion in the respective first and second image areas of the first image signal. A combiner combines the first and second image areas of the second signal to generate a composite image.

[0016] In one application, the first image signal may be a standard definition (SD) image signal which is converted to a high definition (HD) image signal as the second image signal having twice the resolution of the SD signal in both the horizontal and vertical directions. For some of the frames, pixels of the HD signal may also be generated by an adaptive processing technique.

[0017] The following detailed description, given by way of example and not intended to limit the present invention solely thereto, will best be appreciated in conjunction with the accompanying drawings, in which like reference numerals denote like elements and parts, wherein:

[0018] FIG. 1 is a block diagram of an illustrative television receiver in accordance with the invention;

[0019] FIGS. 2A-2B and 3 are diagrams illustrating pixel data creation in accordance with the invention;

[0020] FIG. 4 is a block diagram of the distortion corrector of FIG. 1;

[0021] FIG. 5 is a graph illustrating a method of determining a motion vector for an image;

[0022] FIGS. 6-8 illustrate data storage within a resolution creating memory;

[0023] FIG. 9 is a flow chart depicting a routine for creating high resolution frames;

[0024] FIG. 10 is block diagram of an illustrative distortion corrector;

[0025] FIG. 11 is a block diagram of another embodiment of a receiver in accordance with the invention;

[0026] FIG. 12 is a block diagram of an illustrative resolution converter within the receiver of FIG. 11;

[0027] FIG. 13 depicts data storage locations of a resolution creating memory of the converter of FIG. 12;

[0028] FIG. 14 is a flow diagram depicting a routine for writing image data within a memory of the resolution converter;

[0029] FIG. 15 illustrates storage of image data in the resolution converter in accordance with a motion vector of an image;

[0030] FIG. 16 is a flow diagram illustrating a routine for reading image data from a resolution converting memory within the resolution converter;

[0031] FIG. 17 is a flow diagram depicting a process for generating high definition pixel data;

[0032] FIG. 18 is a block diagram of a circuit for generating high definition image data from standard definition image data employing prediction coefficients;

[0033] FIG. 19 is a diagram illustrating class taps and predictive taps;

[0034] FIG. 20 is a block diagram of a learning device for determining prediction coefficients;

[0035] FIG. 21 illustrates data storage within a resolution creating memory with an enlarged access area;

[0036] FIG. 22 illustrates frame to frame movement of an image;

[0037] FIGS. 23 and 24 illustrate storage of pixel data of the image of FIG. 22;

[0038] FIG. 25 illustrates horizontal scanning lines of a CRT display;

[0039] FIG. 26 illustrates undersampling of an image signal in the vertical direction; and

[0040] FIG. 27 is a block diagram of another resolution converter in accordance with the invention.

[0041] Referring to FIG. 1, a simplified block diagram of a television receiver 10 in accordance with the present invention is shown. Receiver 10 converts an input television signal to a digital signal, corrects the digital signal for aliasing distortion while enhancing its resolution, and then converts the distortion-corrected digital signal back to an analog signal for display.

[0042] Receiver 10 includes a tuner 1 for demodulating an input television broadcast signal of a selected band, which is received via an antenna or transmission line. The demodulated signal is applied to low pass filter (LPF) 2, which operates as an aliasing distortion prefilter to filter out high frequency components of the television signal. The filtered output signal of LPF 2 is applied to analog to digital (A/D) converter 3 where it is sampled and digitized. The digitized signal is then provided to distortion corrector 4 which reduces or essentially eliminates aliasing distortion in the vertical direction of the digitized television signal. An image processing circuit such as a Y/C separator 5 is employed between distortion corrector 4 and D/A converter 6 (or alternatively between tuner 1 and LPF 2). Y/C separator 5 separates luminance from chrominance in the output signal from distortion corrector 4. It is noted that in the configuration shown, a composite signal (luminance plus chrominance) is supplied to A/D converter 3, and sampling is performed in accordance with the frequency of the chrominance subcarrier. If, in the alternative, Y/C separation of the television signal is performed prior to the signal being applied to A/D converter 3, then sampling would be performed at a frequency on the order of 13.5 MHz. In either case, D/A converter 6 converts the distortion-corrected output signal of distortion corrector 4 back to an analog signal to be displayed on CRT 7.

[0043] The reproduction of an original image from an image having aliasing distortion in the vertical direction, i.e., normal to the horizontal scanning lines, will now be described. Referring to FIG. 2A, the principal of an aliasing distortion removal method performed within distortion corrector 4 is illustrated. The image signal variation in the vertical direction within a given column C_iof each frame of the image is designated as signals S_Nto S_N+4for frames N to N+4, respectively. Pixels P₁to P_Kof column C_iin each frame are located at the respective positions of conventional horizontal scanning lines SL₁−SL_K. The signal level of each pixel is illustrated horizontally in the figure—for example, in frame N, signal levels V₁−V_kcorrespond to pixels P₁−P_k, respectively.

[0044] In accordance with one embodiment of the present invention, pixels are created at positions in between the conventional horizontal scanning lines to remove aliasing distortion and, optionally, to improve resolution of the displayed image. These inter-scanning line pixels are created in accordance with the motion of the image from frame to frame. To illustrate this concept, reference is first made to frame N, where the signal variation of an original image S_Nalong column C_ifor that frame is shown. Due to the high spatial frequencies within signal S_N, vertical sampling at positions corresponding to the horizontal scan lines is insufficient for reproducing all the image information therein, whereby aliasing distortion would result without the pixel creation method of the invention. That is, the vertically oriented image signal based on merely sampling one frame at a time at the conventional horizontal scan lines would resemble a distorted signal S_N′ (shown as a dashed line), which is markedly different from the original image signal S_N.

[0045] To generate inter-scanning line pixels that recapture the original image, frame to frame motion of the image signal is detected. For frame N, the signal values V₁−V_Kfor the respective pixels are stored in memory. The motion of the image is then detected between frame N and N+1. (Illustrative methods of detecting frame to frame motion will be discussed later). In the example of FIG. 2A, the image moves downward by a distance D₁between frames N and N+1 . For frame N+1, the image is again sampled at the horizontal scanning lines to generate signal values V₁′−V_K′ for pixels P₁−P_kof frame N+1 and those values are stored in memory. In addition, inter-scanning line pixels P_1a, P_2a, . . . P_Kaare created and stored in memory, where pixel P_1ahas the signal value of V₁determined from the frame N sample for pixel P₁, pixel P₂a has the signal value of V₂determined from the frame N sample for pixel P₂, and so forth. The location of pixel P_1ain frame N+1 is the distance D₁from scanning line SL₁, determined based on the motion from frame N to N+1. Likewise, pixel P_2ais located the distance D₁beneath scanning line SL₂.

[0046] The pixel generation process is repeated for frame N+2, where the image signal S_N+2is sampled at the horizontal scan lines to determine signal values V₁″−V_K″ for respective pixels P₁−P_Kof that frame, and the motion from frame N+1 to frame N+2 is detected. Assuming the image moves down by a distance D₂, pixels P_1b, P_2b, . . . , P_Kbare generated having values of V₁′, V₂′, . . . V_K′ corresponding to the previously determined values stored in memory for frame N+1, and each located a distance D₂below the respective horizontal scan line SL₁. Likewise, frame N+2 contains pixels P_1a, P_2a, . . . , P_Kahaving values of V₁, V₂, . . . V_K, respectively, as previously stored in memory for frame N+1, and each located a distance (D₁+D₂) below SL₁.

[0047] The pixel generation process proceeds for the subsequent frames until a predetermined number of additional pixels are generated. In the example of FIG. 2A, the process continues until frame N+4, which has five times as many pixels in the vertical direction as in frame N. FIG. 2B shows the pixels between scan lines SL₁and SL₂of frame N+4. It is assumed that the image moved down by a distance D₃between frames N+2 and N+3, and by a distance D₄between frames N+3 and N+4. Thus, frame N+4 has a pixel P₁of value V₁″″ corresponding to the level of signal S_N+4at position SL₁, as well as pixels P_1d, P_1c, P_1band P_1aof values V₁′″, V₁″, V₁′ and V₁, respectively, where P_1dis a distance D₄below SL₁, P_1cis a distance (D₃+D₄) below SL₁, and so on. Of course, more or fewer additional pixels can be generated as a function of the resolution desired. In any event, in the present example, all frames subsequent to frame N+4 will have five times as many pixels in the vertical direction as the number of horizontal scan lines.

[0048] Accordingly, pixel generation is accomplished by assuming pixels at locations corresponding to the motion. Practically, however, it is desirable to represent an image with uniformly spaced samples. In the above example, if the inter-frame motion is non-uniform such that the distances D₁−D₄are unequal, the pixels that are created will be non-uniformly spaced. However, by appropriately rounding off the inter-frame motion, pixels of uniform spacing can be derived and stored in memory. Assuming that frame N+4 is thus represented by uniformly spaced pixels in the vertical direction (at five times the number of scanning lines in this example), each frame thereafter can be represented by the same number of uniformly spaced pixels.

[0049] In the example of FIG. 2A, it was assumed that inter-frame motion of the image was strictly in the vertical direction. To remove aliasing distortion and increase the number of pixels in the vertical direction when the original image is moving in both horizontal and vertical directions, horizontal motion should be taken into account as well. To this end, a motion vector is computed from frame to frame as will be discussed more fully below. In addition, different portions of the image within the frame typically move in different directions. Certain embodiments of the invention take such divergent motion into account as well by analyzing different blocks of each frame separately.

[0050] FIG. 3 is a diagram illustrating the general concept of enhancing resolution and removing aliasing by creating pixels based on image data from previous frames and frame to frame image movement. The upper part of the diagram shows frames N to N+3 with image data 101-104, respectively, along common horizontal scan lines. Assuming that the image moves incrementally downward from frame N to N+3, additional pixels are generated in between the scan lines based on the data from the previous frame and the image motion as described above. Thus, frame N+1 is enhanced with pixel data 101 of frame N; pixel data 101, 102 is added to frame N+2 and pixel data 101, 102 and 103 is added to frame N+3 to generate a high resolution image with four times as many pixels in the vertical direction as compared to the original frame.

[0051] Referring now to FIG. 4, a block diagram of an illustrative configuration for distortion corrector 4 is depicted. A frame memory 11 receives input image data applied to the distortion corrector. The input image data is assumed to correspond to an undersampled image signal such that aliasing distortion would normally result if the frames were simply reproduced without any resolution-enhancing pixels being added. Frame memory 11 is composed of a present frame memory 11A for storing image data of a present frame and a preceding frame memory 11B that stores image data of the frame immediately preceding the present frame. A motion detector 12 detects a motion vector that expresses the motion of the present frame with respect to the preceding frame. Such inter-frame motion is detected by a finer amount in the vertical direction than the size of a pixel (or pixel to pixel spacing) within the distorted image. For the present discussion, it is assumed that the image of an entire frame moves together, such that motion from the preceding frame to the current frame is the same for all portions of the each frame. For this case, only one motion vector is detected by detector 12 for each frame. The motion vector is provided to controller 14.

[0052] One method of determining the motion vector is to perform several comparisons between pixel blocks of identical sizes in adjacent frames, e.g., 8×8 or 16 ×16 pixel blocks. Starting with a subject block of the present frame, image characteristics of a number of blocks in the preceding frame (reference blocks) are compared to the subject block. When a reference block in the preceding frame is found with the closest image characteristics to the subject block, it can be concluded that that reference block moved to the position of the subject block, and the motion vector can be determined.

[0053] A subject block is compared to a reference block by computing the absolute value difference between pixel values of the pixels in the subject block to corresponding pixels of the reference block. Thus, the signal value of pixel P₁₁(row 1, column 1) in a subject block is subtracted from the signal value of pixel P₁₁in the reference block, and so forth. The difference value is referred to as the error. The errors of each reference block with respect to the subject block are summed, the reference block with the minimum total error is determined, and the motion vector is determined as being approximately in the direction corresponding to the positional relationship between that reference block and the subject block. More specifically, as shown in FIG. 5, a linear interpolation between the reference blocks is performed to more accurately determine the motion vector. The intersection of a pair of linearly interpolated lines of opposite slope defines the relative position in which the minimum error occurs between the subject block and a hypothetical reference block. The intersection defines the motion vector with respect to the subject block.

[0054] With continuing reference to FIG. 4, a scene change detector 13 is employed to detect a scene change of the television signal by analyzing differences between images from frame to frame. Generally, if the differences between images of the present frame and the preceding frame exceed a predetermined threshold, a scene change is detected. Methods for detecting scene changes are well known in the art. For instance, scene change detector 13 may be configured to analyze errors between subject blocks and reference blocks in a manner similar to determining a motion vector, and finding the minimum error values for the respective subject blocks forming the present frame. If the total sum of the minimum error values for the respective blocks exceeds a predetermined threshold, a scene change is detected and a scene change signal is provided to controller 14.

[0055] Controller 14 controls writing of data from frame memory 11 to a resolution creating memory 15 in accordance with the motion vector supplied from motion detector 12 and the signal from scene change detector 13. Resolution creating memory 15 is capable of storing more image data than the data within one low resolution frame in frame memory 11 so as to store newly created high resolution frames that are sequentially shifted out. Controller 14 controls writing of image data from memory 11 to memory 15 by determining appropriate storage addresses within memory 15 in accordance with the motion vector. High resolution frames are created within memory 15 in a similar manner as was described above in connection with FIG. 2A. Briefly, to create inter-scanning line pixels (resolution-enhancing pixels), data of present frame memory 11A (or from preceding frame memory 11B) is transferred to a high resolution “frame” of memory 15 at the same pixel locations within the frame according to the scanning lines. Image data of the next frame is then written into memory 15 at translated positions according to the motion vector. Controller 14 supplies a relative address pointer to memory 15 to effectuate data transfer to the translated positions, while leaving at least some of the image data of the previous frame stored in memory 15 intact at the same storage locations. Once a frame is created, it is transferred out of memory 15 to vertical low pass filter 16 under the control of controller 14.

[0056] It is noted that following a scene change, it would take the time required to receive and process several low resolution frames for a true high resolution frame to be created, as is apparent from FIG. 2A. Thus, when a scene change occurs, interpolation between low resolution image data may be performed to initially create additional pixels in memory 15.

[0057] FIG. 6 illustrates storage of image data within resolution creating memory 15. Memory 15 is capable of storing image data of P_H′ pixels in the horizontal (scanning line) direction, and of P_V′ pixels in the vertical direction. By way of example, it will be assumed in the following description that P_H′>P_Hand P_V′>4P_V, where P_Hand P_Vare the number of pixels in the horizontal and vertical directions, respectively, for low resolution frames stored within frame memory 11. For this case, resolution creating memory 15 is capable of storing a larger number of pixels in the horizontal direction and more than four times the number of pixels in the vertical direction than the low resolution image. As used hereafter, the term “distorted image” or “distorted data” will be used to refer to the image or data as stored within frame memory 11, which data corresponds to an undersampled image in the vertical direction. As explained previously, however, a distorted reproduced image would only result if the data were read out directly from frame memory 11 and reproduced frame by frame. In the present embodiment, the “distorted” image data is used to create a non-distorted, regenerated image.

[0058] Within resolution creating memory 15, a data storage location can be defined with an absolute address and also with a relative address. FIG. 6 depicts data storage in accordance with absolute addresses, where the absolute addresses encompass all the storage locations of memory 15. The first row and column of the absolute address storage area are designated as row 0 and column 0. As such, a pixel in the (i+1)st row and (j+1)st column is designated with an absolute address of (i,j).

[0059] As shown in FIG. 7, within the absolute address storage array of memory 15 there is defined an access area 15acorresponding to P_H×4P_Vpixels. The location of the upper left point of access area 15ais defined by a relative address pointer R_p, designated as an arrow in FIG. 7. Address pointer R_pmay actually be a codeword supplied by controller 14 to control circuitry (not shown) within memory 15. The address pointer functions to control where incoming image data is to be written. As is the case for the larger absolute address area, access area 15aalso begin with a row 0 and a column 0.

[0060] FIG. 8 illustrates how the image data originating from frame memory 11 is written into access area 15a.In the horizontal direction of the access area, image data of the same number of pixels as in a frame of frame memory 11 is stored, whereas in the vertical direction, data of four times as many pixels are stored. Thus, as shown by the shaded area in FIG. 8, the write of a distorted image from frame memory 11 into access area 15ain the horizontal direction is performed sequentially from the address defined by address pointer R_p, but in the vertical direction it is performed every fourth line. The pixel data in between the scanning lines, e.g., in rows 1-3, 5-7, etc. in the grid of FIG. 8, are created and stored in accordance with the motion vector to generate a high resolution frame in the vertical direction. To realize higher resolution than that of the distorted image, the motion vector is detected in finer units than the pixel size of the frames in frame memory 11.

[0061] The storage of data within resolution creating memory 15 will now be explained in further detail with reference to the flow chart of FIG. 9. When controller 14 receives a signal from scene change detector 13 indicative of a scene change from the preceding frame to the present frame, it resets (clears) all data in memory 15 (step S1). It is noted here that a scene change signal will also be received when the very first frame of data is received by frame memory 11. (Whether the first frame of data is received or a scene change has occurred, the operation by controller 14 is the same.) In step S2, controller 14 causes distorted data of present frame memory 11A to be written into access area 15aevery fourth line as shown in FIG. 8. At this point, gaps are present in access area 15ain between the storage locations corresponding to the scan lines, since these storage locations were previously reset. These gaps are filled by means of interpolation performed by controller 14 in step S3. That is, controller 14 computes interpolated values for the inter-scan line pixels and causes those values to be stored as data within access area 15a.The image data including the interpolated values are then read out from memory 15 in step S4 and forwarded to vertical low pass filter 16.

[0062] In step S5, the data in present frame memory 11A is shifted to previous frame memory 11B, the next frame of data is received and stored within present frame memory 11A, and it is again determined if a scene change has occurred. If not, motion is detected by motion detector 12 (step S6) and a motion vector is computed. Controller 14 then moves the relative address pointer in accordance with the motion (step S7) and data is written into the high resolution frame within memory 15 in accordance with the motion vector. This operation can be executed in a number of ways. One way would be to transfer out the previous data stored within access area 15ainto a temporary buffer memory within controller 14, and then re-write that data back into access area 15a,but shifted in accordance with the motion vector. Following the shift, the data of present frame memory 11A (which has not yet been written into memory 15) would be written into the same access area 15apreviously established, i.e., to the same scanning line positions, and every fourth line. As such, some of the previous data would be overwritten. With this approach, the previous high resolution frame data would be shifted each frame in accordance with the motion, as was illustrated previously in FIGS. 2A and 3 where data from each preceding frame was sequentially shifted in direct correspondence with the motion. (One difference, however, is that no interpolated data was assumed in the methods of FIGS. 2A or 3—gaps in the high resolution frame were assumed until the fifth frame in FIG. 2A and until the fourth frame of FIG. 3.)

[0063] While the above approach of shifting the storage positions of the high resolution data in direct relation to the motion vector will suffice to create a proper high resolution frame, a preferred approach is to maintain the previous data in access area 15ain the same storage locations, while writing the new low resolution data of present frame memory 11A to shifted locations relative to the previous data within the access area. In other words, the data of frame N+1 will be written into storage locations with different absolute addresses than the data of frame N whenever the motion vector for frame N+1 is non-zero. At the same time, the absolute addresses of the previously data in memory 15 remain the same, so the net effect is a shift of the previous data with respect to the new data. This approach is carried out merely by shifting the relative address pointer in correspondence with the motion vector, but in the opposite direction. Thus, the relative addresses of the low resolution (distorted) data will remain the same (e.g., rows 0, 4, etc. as shown in FIG. 8), but the relative addresses of the preceding high resolution frame data will change in direct relation to the motion vector. In essence, the physical access area 15aof memory 15 will shift with each new relative address pointer.

[0064] With each new motion vector, the relative address pointer moves by the same number of pixels as the number of pixels in the horizontal component of the motion vector. In the vertical direction, however, the relative address pointer moves by four times as many high resolution pixels as the number of low resolution pixels in the vertical component of the motion vector (since there are four times as many pixels in the vertical direction of the high resolution frame as in the low resolution frame). Naturally, the relative address pointer is rounded off to the nearest pixel corresponding to the motion vector (or the motion vector is rounded off). Thus, for example, if the vertical component of the motion vector is between ⅛ and ⅜ of the size of an original low resolution pixel, then the relative address pointer will move by one vertical unit. (Alternatives to simple round off of the motion vector will be discussed later.)

[0065] With continuing reference to FIG. 9, the writing of new low resolution (distorted) image data into memory 15 in accordance with the relative address pointer (step S8 ) creates a new high resolution frame, which is then read out to vertical LPF 16 in step S9. If, in step S10 it is determined that low resolution data is no longer being applied to frame memory 11, the routine ends; otherwise, the routine returns to S5 where the process is repeated for each subsequent frame of input image data.

[0066] Accordingly, by repeating steps S5 to S9 several times without a scene change, the interpolated data that filled the gaps between the scanning line samples are sequentially replaced by image samples. Several frames after a scene change, the image frames become true high resolution images, and when the highest frequency in the vertical direction contained within the original image is lower than ½ the frequency corresponding to ¼ of horizontal scanning period of the distorted image, the reproduced image will have no aliasing distortion in the vertical direction.

[0067] Returning to FIG. 4, vertical LPF 16 functions to limit high frequency components in the image data from resolution creating memory 15 by filtering in the vertical direction with a low pass filter. The purpose of such filtering is to prevent aliasing distortion even when the image to be subsequently reproduced is a low resolution image. For example, if only a low resolution display is available, a low resolution image will be displayed, but nevertheless with aliasing distortion removed. Consequently, when the image is subsequently reproduced, the viewer will not perceive an unnatural or blurred moving image. The output signal of vertical LPF 16 is applied to a frame memory 17 which may be have the same storage capacity of access area 15aof memory 15. Frame memory 17 temporarily stores the image data, which is then read out, for example, every fourth line to produce a low resolution image of the same number of lines as the original distorted image but with aliasing distortion removed.

[0068] Accordingly, the above-described method improves image quality and renders it possible to prevent drawbacks that accompany certain signal processing operations such as Y/C separation, noise reduction, and so forth.

[0069] It will be readily appreciated to those skilled in the art that the above-described technique of creating a high resolution image from an undersampled image in accordance with the motion vector from frame to frame, then filtering the same with a vertical LPF and reading out only a low resolution image, will produce an output image with aliasing distortion removed. On the other hand, if the input low resolution image data were merely applied to a vertical LPF, such aliasing distortion removal would not be realized.

[0070] If CRT 6 is capable of displaying a high resolution image by using more horizontal scanning lines than those of the undersampled frames (e.g., four times as many scanning lines), then the image data could be output directly for display. Thus, for this case, vertical LPF 16 would not be utilized, the data would be supplied directly to frame memory 17, and the data would be read out therefrom for display line by line rather than every four lines.

[0071] When the highest frequency in the vertical direction contained in the original image is higher than ½ the frequency corresponding to ¼ of a horizontal scanning period, the sampling theorem would not be satisfied with the above approach of generating four times as many pixels, and the aliasing distortion in the vertical direction would not be completely eliminated. Nevertheless, the method would still be useful in that aliasing distortion will be substantially reduced.

[0072] While in the above embodiment the number of pixels in the vertical direction are increased by a factor of four, in the alternative, more or fewer inter-scan line pixels may be created, either for the purpose of displaying low resolution or high resolution images with reduced aliasing distortion.

[0073] For the above-described distortion corrector 4a,it was assumed that the entire image of any given frame moved together, i.e., in the same direction and by the same distance from frame to frame. In other words, it was assumed that the motion vector for all portions of each frame was the same. That assumption is generally valid for certain applications such as pan and tilt operations. However, in most video applications, different portions of each frame move at different rates and/or in different directions. Embodiments of the present invention to be described below are capable of reducing aliasing distortion while accounting for such independent motion.

[0074] Referring now to FIG. 10, a block diagram of another illustrative embodiment of a distortion corrector in accordance with the invention is shown. Distortion corrector 4bhas the capability of discerning frame to frame motion for multiple portions of each frame, and generating high resolution frames based on the motion. The high resolution frames may be employed either to generate low or high resolution images, with aliasing distortion removed in either case.

[0075] Distortion corrector 4bincludes essentially the same frame memory 11, motion detector 12, scene change detector 13, vertical low pass filter 16 and frame memory 17 as was described above for distortion corrector 4a.Motion detector 12 was described above as operative to determine a motion vector of a present frame subject block of, e.g., 8×8 or 16×16 pixels, from the preceding frame by analyzing differences between the subject block and a plurality of reference blocks of the previous frame. In distortion corrector 4a,the motion vector of one subject block was sufficient to determine the motion of the entire frame image. On the other hand, with distortion corrector 4b,a number of motion vectors are determined for each frame, each corresponding to a different subject block. The motion vectors of the various subject blocks are provided to an area divider 21 which divides the frame area into multiple areas for independent processing operations. The areas are defined based on the motion vectors, differences between adjacent pixels, and by means of other necessary operations such as smoothing, etc. By way of example, if a scene includes a fixed or slow moving object such as a background and a fast moving object as an airplane, area divider 21 would define a first area of the frame as corresponding to the background and a second area corresponding to the airplane. In FIG. 10, area divider 21 is adapted to define two areas per frame; however, it is understood that it may alternatively be configured to divide each frame into three or more areas rather than just two areas.

[0076] Once area divider 21 defines the image areas for the present frame, it controls the read-out of image data from present frame memory 11A by controlling switches 22A and 22B so that only image data from the first area is directed through switch 22A for temporary storage in a memory 23A, and only data from the second area is directed through switch 22B for storage within memory 23B. Area divider 21 also provides a motion vector for a region within the first area to controller 14A and a motion vector for a region within the second area to controller 14B. Scene change detector 13 supplies scene change signals to controllers 14A and 14B. Controller 14A controls writing of data from memory 23A into resolution creating memory 15A by dynamically changing a relative address pointer based on the motion vector in the first area for each frame. Consequently, inter-scanning line pixels are created in positions according to the motion vector in essentially the same manner as described previously. A high resolution frame or frame portion with, e.g., four times as much resolution in the vertical direction as that in frame memory 11A is thereby created in memory 15A. Likewise, image data of the second frame area is transferred from memory 23B to memory 15B under the control of controller 14B in accordance with the second motion vector to produce a high resolution frame or frame portion in memory 15B. That is, each of controllers 14A and 14B operates in conjunction with memories 15A and 15B, respectively, by dynamically determining respective address pointers in the same manner as discussed above for controller 14 and memory 15 of FIG. 4, except that only a portion of the overall image is enhanced and stored within each memory 15A and 15B.

[0077] Data of each high resolution frame created within memories 15A and 15B is transferred to a combiner 24 under the control of controllers 14A and 14B, respectively. Combiner 24 combines the data of the two frame portions to produce image data of a composite frame and provides the same to vertical LPF 16. Thus, the output image data of combiner 24 is similar to that of memory 15 in distortion corrector 4a.Vertical LPF 16 and frame memory 17 perform the same processes as described previously to produce a low resolution image with aliasing distortion removed. Conversely, if a high resolution image is desired, vertical LPF may be bypassed or designed with a different frequency response as discussed earlier.

[0078] While distortion correctors 4aand 4bhave been described as being operative to create inter-scan line pixels in the vertical direction based on the motion vector (i.e., the motion vector corresponding to both vertical and horizontal motion of the image), image data for additional pixels in the horizontal direction can also be created. That is, high resolution pixels can also be created in memory 15 (or memories 15A, 15B) at positions in between low resolution pixel positions in the horizontal direction in an analogous manner as was described for the vertical direction. As a result, aliasing distortion can further be eliminated in the horizontal direction and higher resolution can be obtained horizontally as well.

[0079] A special case exists in which an access area swells out of the storage area of resolution creating memory 15 (or 15A, 15B) in accordance with the position of the relative address pointer. For this case, the data of the swollen out region can be retained by expanding the storage area of the resolution creating memory into a neighboring area. Assuming that X≧P_H′, and Y≧P_V′, when a position expressed by an absolute address (X,Y) is included in the access area, the position is retained at an absolute address of (mod(X, P_H′), mod(Y, P_V′)) in the storage area of memory 15 (where mod (a, b) expresses the residue when “a” is divided by “b”). Another way of retaining the swollen out data is to relocate the access area including the most recent data retained therein, by controller 14 transferring all the data to a buffer memory and then rewriting it to a new access area using a new address pointer.

[0080] In the embodiments discussed above, high resolution frames are created via the assumption of inter-scanning line pixels at positions corresponding to the motion vectors. The high resolution frames are then employed to generate either low or high resolution images with aliasing distortion reduced (or eliminated). Another method of performing such image regeneration is as follows: it is assumed that an original image with no aliasing distortion is designated as P and a distorted (undersampled) image based thereon as P′. The relation between the two is expressed as:

[0081] P′=f(P)  (1)

[0082] where f( ) denotes a function that subsamples an image in the parentheses.

[0083] Likewise, an inverse function of f( ) may be expressed as g( ), so that:

[0084] P=g(P′)  (2).

[0085] Therefore, the regeneration of the original image P from the distorted image P′ can be realized theoretically if the function go is found, which renders it possible to reduce aliasing distortion without the necessity of detecting motion of the distorted image.

[0086] The function g( ) can be obtained via a learning process in which the original image is considered to contain certain teacher-data and consecutive frames of the distorted image P′ are considered to contain learning-data. The inter-scanning line gaps of the distorted image can then be filled based on the function g( ), whereby the original image P can be regenerated.

[0087] Aside from removing aliasing distortion, the present invention has other applications such as for producing enlarged images with enhanced resolution, or for converting a low or standard resolution image (hereafter referred to as a SD image) into a high resolution image (hereafter referred to as an HD image).

[0088] With reference now to FIG. 11, there is shown a block diagram of an illustrative embodiment of a television receiver 100 that converts a television signal containing an SD image into an HD image for display. Receiver 100 differs from receiver 10 of FIG. 1 by employing a resolution converter 34 in place of distortion corrector 4 and a high resolution CRT 36 in place of CRT 7. The other components are the same as those of receiver 10, with a Y/C converter (not shown) typically employed either between resolution converter 34 and D/A converter 6 or between LPF 2 and A/D converter 3.

[0089] FIG. 12 is a block diagram of an embodiment of resolution converter 34. Resolution converter 34 is similar in many respects to distortion corrector 4bof FIG. 10, in that it operates to divide the area of a low resolution frame in accordance with different motion in respective areas, creates a high resolution image for each area and combines the high resolution images for each area to produce a composite image. Resolution converter 34 is configured to enhance resolution in both horizontal and vertical directions by creating pixels in positions between adjacent horizontal pixels and also between adjacent vertical pixels of the SD image based on the motion vectors for each frame. Converter 34 may generate some of the high resolution pixels by a classification and adaptive processing technique to be discussed in detail later.

[0090] Resolution converter 34 includes frame memory 11, motion detector 12 and scene change detector 13 as described previously, as well as area divider 21′ which is the same as area divider 21 of FIG. 10 except that divider 21′ divides each frame into M frame areas rather than just two areas. Each one of M high resolution object generators functions to generate a high resolution object for a respective one of the frame areas defined by area divider 21′. Each object generator 41i(i=1, 2, . . . M) includes switch 22i,buffer memory 23i,controller 14i,resolution creating memory 15i,write flag memory 42iand pixel generator 43i.

[0091] Each write flag memory 42istores the same number of write flags as the number of pixels stored in associated resolution creating memory 15i.The write flag is a one bit flag designating whether pixel data of an SD image is stored in a corresponding address of memory 15i(e.g., the flag is a 1 if pixel data is stored in the address and 0 if it is not). The write flag is controlled by controller 14ito be set/reset.

[0092] Referring to FIG. 13, each resolution creating memory 15iis designed with a storage area capable of storing pixel data of P_H′ pixels in the horizontal direction and P_V′ pixels in the vertical direction, with P_H′≧2P_Hand P_V′≧2P_V, where P_Hand P_Vare the number of pixels in the horizontal and vertical directions, respectively, of an SD frame stored within frame memory 11. In this example, memory 15iis capable of storing pixel data of more than twice as many pixels in the vertical direction and more than twice as many pixels in the horizontal direction as the SD image. Memory access area 15aiis comprised of storage locations to store pixel data of 2P_H×2P_Vpixels.

[0093] A procedure for writing a frame of SD data into resolution creating memory 15iwill now be described with reference to the flow chart of FIG. 14. In step S11, if scene change detector 13 detects either a scene change or the reception of the first frame of data, a scene change signal is provided to each of controllers 14₁−14_M. Each controller 14_ithen clears the associated memory 15_iin step S12 (erasing previous frame pixel data) and resets all write flags in write flag memory 41_ito an initial value (step S13). Distorted SD image data of the associated frame area of present frame memory 11A is then written into memory 15i(step S14) by closing switch 22iat appropriate times in a frame data readout operation from frame memory 11A. Thus, only the image data within the associated area as defined by area divider 21′ is transferred through switch 22iand memory 23i.In particular, SD data is written into the access area of memory 15iat every other pixel position in both horizontal and vertical directions. Thus, for example, it is assumed that SD frames are arranged with M pixel rows (rows 1 to M) by N pixel columns (columns 1 to N), and the access area of memory 15iis a storage array corresponding to 2M rows (rows 1 to 2M) by 2N columns (columns 1 to 2N). Then, if data of pixel 1,1 corresponding to row 1, column 1 of the SD frame is transferred to object generator 41i,it will be initially stored in the storage location of the access area corresponding to the HD pixel of row 2, column 2 of the access area. Preferably, that storage location will also be row 2, column 2 in the storage array, to facilitate pixel creation. Similarly, SD data of pixel 2,2 of the SD frame will be initially stored in row 4, column 4 of the access area, and so forth. Write flags are concurrently set for each of the HD storage locations in which SD data is thus stored. The image data is read out from memory 15iby pixel generator 43iat this point, but is not erased from memory 15ibecause it will be used to form part of the next high resolution frame (in the absence of a scene change).

[0094] If, in step S11, no scene change signal is received, then in step S16, each controller 14ireceives the motion vector from area divider 21′ corresponding to the motion in the associated image area of the present frame. Controller 14icomputes a new relative address pointer R_pcorresponding to the motion vector and supplies the same to resolution converting memory 15i.The routine then proceeds to step S14 where the distorted image data of the present frame is written into memory 15iin accordance with the relative address pointer just determined, whereby data previously stored in the preceding frame of memory 15iis shifted with respect to the new data (since the data of the previous frame remained stored at the same locations in accordance with previous address pointers while data of the present frame is stored in accordance with the new address pointer). As explained previously, some of the previously stored data is typically overwritten in this step. Write flags are then set for the storage locations in which new data is stored in step S15 (unless they were already set previously).

[0095] Accordingly, as the write process proceeds frame by frame in each resolution creating memory 15i,an HD image having twice as much resolution as the corresponding SD image in both horizontal and vertical directions gradually approaches to completion.

[0096] To further illustrate the creation of an HD image, reference is made to FIG. 15 which illustrates inter-frame movement of a small, solid triangular object 47 and associated storage of pixel data representing the same. The left hand side of the figure shows an array of SD pixels P_SDsuperimposed with object 47 while the right hand side depicts storage locations MC within access areas 15a_Nto 15a_(N+3)for frames N to (N+3), respectively. In the low resolution image, SD pixels P_SDwithin the bounded region of object 47 represent the object by, e.g., having more or less luminance and/or being of a different color than the neighboring pixels. In any event, object 47 is assumed to be represented by pixels a, b, c and d in frame N, by pixels f, g in frame (N+1), pixels h, i, j, k in frame (N+2) and by pixels 1, m in frame (N+3), where the pixels change from frame to frame due to the inter-frame motion.

[0097] In accordance with the above-described method, SD pixel data is written into access areas 15a_Nto 15a_(N+3)in the shaded regions in alternating storage locations. In frame N, storage locations (a′ to e′) are filled with pixel data corresponding to SD pixels (a to e). Between frames N and N+1, the object moved upwards by ½ the SD pixel spacing (spacing between midpoints of adjacent SD pixels) so that relative address pointer R_pof frame N+1 moves down by one HD pixel storage location as shown, and the pixel data for pixels f and g are written. Meanwhile, the absolute storage locations storing pixel data (a′-e′) remain the same in frame N+1. Thus, it is seen that while pixels b and g are at the same location in the SD image, the corresonding “pixels” b′ and g′ are shifted with respect to one another in access area 15a_(N+1)representing the HD image. The process continues with address pointer R_pmoving from its previous position in the opposite direction as the object motion, until a complete HD image is formed in frame (N+3). Note that it would take more frames to generate a complete HD image based solely on generating HD pixel data in accordance with the motion vector if the object were to remain still or if the motion is slight from frame to frame.

[0098] FIG. 16 is a flow chart illustrating a routine for reading image data from high resolution object generator 41i.The reading routine follows the routine of FIG. 14 each time new image data is written into memory 15i,i.e., each time writing of data for a particular frame is completed. In step S21, controller 14ireads image data from storage locations of access area 15iand supplies the data to pixel generator 23. In step S22, pixel generator 23 determines, based on the write flags in write flag memory 42i,whether or not image data is stored for a particular high resolution pixel corresponding to a particular address. For instance, considering FIG. 15, assuming that frame N corresponds to the first frame after a scene change, the storage locations c′ and d′ contain stored image data but the storage location 48 in between c′ and d′ is empty. As such, the write flag for c′ and d′ would be set to 1 whereas that for storage location 48 is at 0. Thus, prior to displaying the image data for a current frame such as frame N, pixel data needs to be created for HD pixels corresponding to the empty storage locations as 48.

[0099] Therefore, in step S23, image data is generated by pixel generator 43iif the storage location under consideration is empty as indicated by the associated write flag. The image data may be generated in various ways, such as by interpolating between neighboring pixels or by a more complex prediction technique such as a classification and adaptive processing method to be described below. In any case, once the image data is created for the pixel under consideration (or read directly from memory 15iif already stored therein), the pixel data for that pixel is outputted from object generator 41iin step S24 and supplied to combiner 24. If all image data of the high resolution frame under consideration is determined to have been read in step S25, the reading process for that frame is complete. Otherwise, the routine returns to step S21 to read data from the next storage location and to once again create pixel data if necessary.

[0100] FIG. 17 illustrates a simplified flow chart for the process step 23 of generating image data for pixels corresponding to empty storage locations, whether by interpolation or by a prediction technique. In step S31, it is determined based on the write flags, which neighboring pixels to the pixel under consideration contain stored pixel data within memory 15i.Then, in step S32, data is read from the storage locations in access area 15aicorresponding to those neighboring pixels. Pixel data for the pixel under consideration is then generated in step S33 based on the neighboring pixel data.

[0101] If a high resolution frame in resolution creating memory 15ihas empty storage locations as described above, a simple interpolation technique to generate pixel data for the empty storage locations may be insufficient to recapture high frequency components of the original image. The present applicant has previously proposed an image converter device which converts an SD image into an HD image containing a high frequency component that was not contained in the SD image. See U.S. Pat. No. 5,517,588 entitled DIGITAL DATA CONVERSION EQUIPMENT AND A METHOD FOR THE SAME, which issued on May 14, 1996. Accordingly, the same or similar adaptive process can be used by pixel generator 43ito generate pixel data for the empty storage locations. That is, the adaptive process can be employed to create “fill-in” pixels in addition to those created by means of assuming pixels at locations corresponding to the detected image motion. An illustrative adaptive process to implement such pixel creation will now be described.

[0102] The adaptive process determines a predicted value of a pixel of an HD image based on linear coupling between an SD image and a predetermined estimation coefficient. For example, a predicted value E[y] of a pixel value y of an HD pixel to form an HD image can be determined by utilizing a linear combination model. This model is defined by a linear combination of pixel values of SD pixels (hereinafter referred to as learning data), x₁, x₂, . . . , and predetermined prediction coefficients, w₁, w₂, . . . For this case, the predicted value E[y] is expressed as follows:

[0103] E[y]=w₁x₁+w₂x₂+  (3)

[0104] To generalize the above, a matrix Y′ is assumed to be composed of a set of the predicted values E[y], for y=Y₁to Y_n. That is, matrix Y′ is defined as the product of a matrix W, which is a set of prediction coefficients w, and a matrix X, which is a set of learning data. An observation equation is thus obtained as follows:

[0105] XW=Y′  (4)

[0106] where, $X=\left(\begin{array}{cccc}{X}_{11}& X_{12}& \cdots & X_{1n}\\ X_{21}& X_{22}& \cdots & X_{2n}\\ ⋮& ⋮& ⋮& ⋮\\ X_{\mathrm{m1}}& X_{\mathrm{m2}}& \cdots & X_{\mathrm{mn}}\end{array}\right),W=\left(\begin{array}{c}\begin{array}{c}\begin{array}{c}{W}_1\\ W_2\end{array}\\ ⋮\end{array}\\ W_n\end{array}\right),Y^{\prime }=\left(\begin{array}{c}\begin{array}{c}\begin{array}{c}E\left[y_1\right]\\ E\left[y_2\right]\end{array}\\ ⋮\end{array}\\ E\left[y_n\right]\end{array}\right).$

[0107] One method of finding a predicted value E[y] that is close to an HD pixel value y entails applying the least square method to the observation equation (4). For this case, matrix Y′ is considered as the sum of a matrix Y and a matrix E, where matrix Y is composed of a set of real HD pixel values (to be used as teacher data) and matrix E is composed of a set of residual “e” of the predicted values E[y]. Thus, the following residual equation is derived from equation (4):

[0108] XW=Y+E

[0109] where, $\begin{array}{cc}E=\left(\begin{array}{c}\begin{array}{c}\begin{array}{c}{e}_1\\ e_2\end{array}\\ ⋮\end{array}\\ e_m\end{array}\right),Y=\left(\begin{array}{c}\begin{array}{c}\begin{array}{c}{y}_1\\ y_2\end{array}\\ ⋮\end{array}\\ y_n\end{array}\right).& \left(5\right)\end{array}$

[0110] The square root error is defined as the sum of the squares of the residuals, that is, $\begin{array}{cc}\sum _{i=1}^m{e}_i^2& \left(6\right)\end{array}$

[0111] Thus, the prediction coefficient w_ifor obtaining a predicted value E[y] close to an HD pixel value can be found by minimizing the square root error of equation (6).

[0112] When the square root error is differentiated by the prediction coefficient w_i, if the result is 0, the value for w_iwhich satisfies the following equation (7) is the optimum value for finding a predicted value E[y] close to the HD pixel value: $\begin{array}{cc}{e}_1\frac{\partial e_1}{\partial w_i}+e_2\frac{\partial e_2}{\partial w_i}+\cdots +e_m\frac{\partial e_m}{\partial w_i}=0\left(i=1,2,\dots ,n\right)& \left(7\right)\end{array}$

[0113] When equation (6) is differentiated by the prediction coefficient w_i, the following equation (8) is obtained: $\begin{array}{cc}\frac{\partial e_1}{\partial w_i}=X_{\mathrm{i1}},\frac{\partial e_1}{\partial w_2}=X_{\mathrm{i2}},\dots ,\frac{\partial e_i}{\partial w_n}=X_{in},\left(i=1,2,\dots ,m\right)& \left(8\right)\end{array}$

[0114] From equations (7) and (8), the following equation (9) is derived: $\begin{array}{cc}\sum _{i=1}^m{e}_i{x}_{\mathrm{i1}}=0,\sum _{i=1}^m{e}_i{x}_{\mathrm{i2}}=0,\cdots \sum _{i=1}^m{e}_i{x}_{in}=0& \left(9\right)\end{array}$

[0115] When the relations among the learning data x, prediction coefficients w, teacher data y and residuals e are considered, a set of prediction equations (10) can be obtained as follows: $\begin{array}{cc}\begin{array}{c}\begin{array}{c}\begin{array}{c}\left(\sum _{i=1}^m{x}_{\mathrm{i1}}{x}_{\mathrm{i1}}\right)w_1+\left(\sum _{i=1}^m{x}_{\mathrm{i1}}{x}_{\mathrm{i2}}\right)w_2+\cdots +\left(\sum _{i=1}^m{x}_{\mathrm{i1}}{x}_{in}\right)w_n=\left(\sum _{i=1}^m{x}_{\mathrm{i1}}{y}_i\right)\\ \left(\sum _{i=1}^m{x}_{\mathrm{i2}}{x}_{\mathrm{i1}}\right)w_1+\left(\sum _{i=1}^m{x}_{\mathrm{i2}}{x}_{\mathrm{i2}}\right)w_2+\cdots +\left(\sum _{i=1}^m{x}_{\mathrm{i2}}{x}_{in}\right)w_n=\left(\sum _{i=1}^m{x}_{\mathrm{i2}}{y}_i\right)\end{array}\\ ⋮\end{array}\\ \left(\sum _{i=1}^m{x}_{in}{x}_{\mathrm{i1}}\right)w_1+\left(\sum _{i=1}^m{x}_{in}{x}_{\mathrm{i2}}\right)w_2+\cdots +\left(\sum _{i=1}^m{x}_{in}{x}_{in}\right)w_n=\left(\sum _{i=1}^m{x}_{in}{y}_i\right)\end{array}& \left(10\right)\end{array}$

[0116] The number of equations in equation set (10) correspond to the number of prediction coefficients w. The optimum prediction coefficients w can be obtained by solving eqns. (10), which can be solved by a conventional technique, for example, by utilizing Gauss-Jordan's erasing method. It is noted that for eqns. (10) to be solvable, a matrix composed of the prediction coefficients w should be a regular matrix.

[0117] Accordingly, with the above-described adaptive process, optimum prediction coefficients w are obtained for the purpose of deriving an optimum predicted value E[y] that is close to an HD pixel value (i.e., close to an HD pixel value which would exist if an HD signal was originally received by the television receiver rather than an SD signal). The adaptive process is different from an interpolation process in that a high frequency component of an original image, absent from an SD image but contained in an HD image, can be restored. The adaptive process, as far as only equation (1) is concerned, is similar to an interpolation process that uses an interpolation filter. For the adaptive process, however, the prediction coefficient corresponding to a tap coefficient of the interpolation filter can be obtained by learning using teacher data. Consequently, the high frequency component contained in the HD image can be restored to readily obtain a high resolution image.

[0118] Turning now to FIG. 18, a block diagram of an image converter device 200 for converting an SD image to an HD image is shown. Device 200 may be used as part of pixel generator 43ito generate HD pixel data when necessary, i.e., whenever an HD pixel storage location in the resolution creating memory is empty. An input SD image signal is applied to both an adaptive processor 204 and a classification circuit 201, the latter being comprised of a class tap generator 202 and a class determining circuit 203. In classification circuit 201, an HD pixel of which a predicted value is to be found in the adaptive process (hereinafter referred to as a marked-pixel) is classified into a predetermined class based on characteristics of a set of SD pixels in a close, predetermined positional relationship to the marked pixel. The pixel values for this set of SD pixels associated with the marked HD pixel are referred to as the class taps of the marked pixel.

[0119] Class taps based on the input SD image are extracted within class taps generator 202 and provided to class determining circuit 203, which detects a pattern of the set of SD pixels forming class taps for each marked pixel. The pattern is based on the characteristics of each of the pixels in the set, which are a function of the pixel values. For instance, one pattern may correspond to evenness of the pixels, a second may correspond to pixel values increasing in an upper right direction, a third may correspond to pixel values increasing in a lower left direction, and so forth. A value previously allotted to the detected pattern is then supplied to adaptive processor 204 as indicative of the class of the marked pixel.

[0120] FIG. 19 illustrates the positional relationship of class taps to associated HD pixels. It is assumed that an HD image is composed of the pixels shown marked as x's and that a corresponding SD image is composed of pixels marked as O's. The shown SD image thus contains one fourth the number of pixels as the HD image. The spacings between the midpoints of the HD columns and between the HD rows are half that for the SD columns and rows. In FIG. 19, an SD pixel at a position (i+1) from the left (SD column i+1) and (j+1) from the top (SD row (j+1)) is expressed as X_ij. Similarly, an HD pixel positioned in HD column (i′+1) and HD row (j′+1) is expressed as Y_i′j′. Thus, for example, the position of the SD pixel X_ijand the HD pixel Y_(2i)(2j)coincide with one another. (It is noted here that “class taps” and “predictive taps” are actually image data values for particular pixels. However, for simplicity in the description, the “taps” will be described as being the pixels themselves, and vice versa).

[0121] To illustrate how class taps are defined, a marked pixel is assumed to be HD pixel Y₄₄, which has the identical position as SD pixel X₂₂. The class taps for this marked pixel are the closest nine SD pixels within a 3×3 SD pixel square centered about the marked HD pixel Y₄₄. Thus, in FIG. 19, the class taps are the SD pixels inside the region defined by boundary T_c, that is, SD pixels X₁₁, X₂₁, X₃₁, X₁₂, X₂₂, X₃₂, X₁₃, X₂₃, X₃₃. Class generator 202 extracts the pixel values for these SD pixels as the class taps for the marked pixel. For the case in which the marked pixel is adjacent to an HD pixel that coincides with an SD pixel, the same class taps as for the coinciding HD pixel can be defined. Thus, for instance, if the marked pixel is either Y₅₄, Y₅₅, Y₄₅, etc. then the class taps for any of these HD pixels may be the same as those for “coinciding” HD pixel Y₄₄. It is also possible to form different class taps for non-coinciding pixels such as Y₅₄, Y₅₅or Y₄₅.

[0122] In classification circuit 203, the pattern of the class taps of a marked pixel are detected (which class taps are provided by class taps generator 202). In other words, the pattern such as evenness, etc., of the nine SD pixels in close positional relationship to the marked pixel are detected to ascertain which class should be assigned to the marked pixel. A pattern value corresponding to the detected pattern is then outputted as the class of the marked pixel, and supplied to the address terminal (AD) of a coefficients read only memory (ROM) 207 in adaptive processor 204.

[0123] Ordinarily, 8 bits or the like are allotted to a pixel which forms an image. Assuming that 8 bits are allotted to an SD pixel, for example, if nine SD pixels are employed for the class taps as in the example of FIG. 19, the number of possible pixel values per class tap is as high as (2⁸)⁹. Consequently, the ability to perform pattern detection processing operations at high speeds is inhibited.

[0124] Therefore, before classification is performed, it is desirable to decrease the number of bits allotted to an SD pixel of a class tap. For instance, adaptive dynamic range coding (ADRC) may be executed to implement such bit reduction. As a first step in the ADRC process, a pixel having the maximum pixel value out of nine SD pixels forming a process block (hereinafter referred to as a maximum pixel) and a pixel having the minimum pixel value in the process block (hereinafter referred to as a minimum pixel) are detected. The difference DR between the pixel value MAX of the maximum pixel and the pixel value MIN of the minimum pixel is then computed. The value DR is designated as a local dynamic reference value of the process block, and the respective pixel values forming the process block are quantized again with a smaller number K of bits for each pixel than the originally allotted number of bits. In other words, the pixel value MIN of the minimum pixel is subtracted from respective pixel values forming the process block, and the respective subtraction results are divided by DR/2K. As a result, respective pixel values composing the process block can be expressed by K bits. Thus, for example, when K=1, the maximum number of patterns of nine SD pixels is (2¹)⁹. Accordingly, the maximum number of patterns is dramatically decreased as compared to when ADRC processing is not implemented.

[0125] With continuing reference to FIG. 18, the adaptive process is performed within adaptive processor 204, which includes predictive taps generator 205, predicted value calculator 206 and prediction coefficients ROM 207. In predictive taps generator 205, data of a number of SD pixels are extracted which are in a predetermined positional relationship to the marked pixel. These extracted pixels are provided as predictive taps x1, x2, . . . to predicted value calculator 206, which determines a predicted value for an HD pixel based on prediction coefficients and the predictive taps.

[0126] The predictive taps correspond to pixels having high positional correlation with the marked HD pixel. By way of example, if the marked pixel is pixel Y₄₄as shown in FIG. 19, and class taps within boundary T_care formed as explained previously, then predictive tap generator 205 may determine the predictive taps as the block of 5×5 SD pixels falling within the area enclosed by boundary T_p, i.e, SD pixels X₀₀to X₄₄. If the marked pixel is a pixel adjacent to Y₄₄such as pixel Y₅₄, Y₄₅or Y₅₅, then the same predictive taps are formed as those for pixel Y₄₄, i.e., corresponding to pixels X₀₀to X₄₄. It is possible, however, to define different predictive taps when a non-coinciding pixel such as Y₄₅, Y₅₄or Y₅₅is the marked pixel.

[0127] Prediction coefficients ROM 207 stores prediction coefficients which are found by learning performed beforehand by classes. When ROM 207 receives a class supplied from classification circuit 203, ROM 207 reads prediction coefficients stored in an address therein corresponding to the supplied class, and supplies the prediction coefficient(s) to predicted value calculator 206.

[0128] Accordingly, predictive taps corresponding to a marked-pixel and the prediction coefficient(s) concerning the class of the marked-pixel are both supplied to calculator 206. Within calculator 206, the operation in accordance with equation (3) above is performed using prediction coefficients w1, w2, . . . received from ROM 207 and SD pixel data x1, x2, . . . which form the predictive taps from predictive tap generator 205. As a result, the predicted value E[y] of the marked-pixel y is determined, and it is outputted as a pixel value for an HD pixel. The process is repeated by designating each HD pixel as a marked-pixel, and when all HD pixels have been so designated and predicted values derived therefor, a complete SD image is converted into an HD image.

[0129] Referring now to FIG. 20, there is shown a block diagram of a learning device 210 that performs a learning process for calculating prediction coefficients to be stored in ROM 207 of FIG. 18. HD image data which is to be the teacher data in the learning process is supplied to both a thinning circuit 211 and a teacher data sampling circuit 146. In thinning circuit 211, the number of pixels of the HD image is decreased by thinning such that the HD image is converted into an SD image. The number of HD image pixels is halved in both the horizontal and vertical directions to thereby form the SD image. The SD image is supplied to classifying circuit 212 and predictive tap generator 145. It is noted, rather than forming the SD image from the HD image, an SD image may be applied directly to classification circuit 212 from an SD camera corresponding to an HD image from an HD camera.

[0130] In classification circuit 212 or predictive taps generator 145, the same processes as those performed in classification circuit 201 or predictive taps generator 205 of FIG. 18 are performed, whereby the class of a marked-pixel or the predictive taps are outputted, respectively. The classes outputted by classifying circuit 212 are applied to the address terminals (AD) of both predictive taps memory 147 and teacher data memory 148. The predictive taps outputted from predictive tap generator 145 are applied to predictive taps memory 147 where the taps are stored at addresses corresponding to the classes supplied from classifying circuit 212.

[0131] In teacher data sampling circuit 146, HD pixels to be marked-pixels in classification circuit 212 and predictive tap generating circuit 145 are extracted out of the HD image supplied thereto. The extracted taps are stored in teacher data memory 148 as the teacher data along with the computed class at a common address location. The process is repeated for all HD pixels of the HD images inputted to device 210 for the purpose of learning. Since predictive tap memory 147 and teacher data memory 148 are each configured to store a plurality of types of information at the same address location, a plurality of learning data x and teacher data y to be classified in the identical class can be stored at essentially the same address location.

[0132] Calculator 149 then reads predictive taps as learning data or HD pixel data as teacher data stored at the same address locations from predictive tap memory 147 or teacher data memory 148, respectively. Based on this data, calculator 149 computes prediction coefficients, e.g, by the least square method which minimizes error between the predicted values and the teacher data. In other words, in calculator 149, the above-described prediction equations (10) are formed by classes, and prediction coefficients are obtained in solving these equations. The prediction coefficients are then stored at the address locations corresponding to the classes in coefficient ROM 207 of FIG. 18. It is noted that independent equations (10) are solved for each “non-coinciding” HD pixel such as pixel Y₄₅, Y₅₄or Y₅₅in FIG. 19 and for each “coinciding” pixel Y₄₄even though the predictive taps are the same.

[0133] In the example of FIG. 19, nine class taps and 25 predictive taps were formed for each marked HD pixel, and the classification and adaptive processes (hereinafter referred to as the classification adaptive process) were performed accordingly. However, for marked HD pixels near the edge of an image area, the assumption of nine class taps and 25 predictive taps is no longer valid. Thus, it is desirable to form class and predictive taps of different arrangements for those HD pixels, and to compute predictive coefficients in a similar manner as that described above to cover these special cases.

[0134] Accordingly, the above-described classification-adaptive process is employed to supplement pixel creation in resolution creating memory 15iwhen HD pixels are not otherwise created in accordance with the associated motion vector. For instance, following a scene change, it will take several frames of motion to generate an HD image from an SD image based on motion alone. Hence, in the initial frames following a scene change, the classification adaptive procedure may be employed often.

[0135] As mentioned previously in connection with the embodiment of FIG. 4, in order to practically create pixels at positions corresponding to image motion, each motion vector may be rounded off to a distance corresponding to the spacing between midpoints of adjacent high resolution pixels. Thus, for example, for the embodiment in which HD pixel data is generated from SD pixel data, if the motion vector of any given frame corresponds to motion of between 0.5 and 1.5 HD pixel spacings in the vertical (or horizontal) direction, the motion vector may be rounded off to one HD pixel spacing in the vertical (or horizontal) direction and the address pointer moved accordingly to write new SD data into the memory. As an alternative, the address pointer can be caused to move only if the motion vector is approximately equal to, or within a predetermined range of, an integral multiple of an HD pixel interval (i.e., an integral multiple of ½ the SD pixel interval). In this case, there will be many instances where there is no change in the address pointer, leading to more frequent empty addresses for HD pixel data in resolution creating memory 15i.Thus, the empty addresses can be filled by interpolation or via the classification adaptive process.

[0136] As another alternative to rounding off the motion vector to create high resolution pixel data in accordance therewith, a different approach is to create an even higher resolution storage array (enlarged access area) within resolution creating memory 15 or 15ito reduce the round off error of the motion vectors. Then, when the yet higher resolution frame is generated, pixel data can be read out by skipping some of the data, e.g., by skipping alternate rows and alternate columns, to generate a high resolution image, but with not quite as high resolution as could be realized.

[0137] To illustrate the latter approach, the aforedescribed resolution converter 34, which is designed to convert an SD image into an HD image, can be modified by employing a resolution converting memory 15_i′ with four times as many pixel storage locations as the SD image in both the horizontal and vertical directions, i.e., with 16 times as much pixel density as compared to the SD image. This approach is illustrated in FIG. 21, in which memory 15_i′ is designed with P_H′>=4P_Hcolumns and P_V′>=4P_Vrows, where P_Hand P_Vare the number of pixels in horizontal and vertical directions, respectively, of an SD frame. Access area 15a_iis designed with 4P_H×4P_Vpixels. Relative address pointer R_pcontrols the precise location of access area 15a_iwithin the larger memory area of memory 15_iin essentially the same manner as explained previously.

[0138] SD pixels are written every fourth position in access area 15a_i′ in both horizontal and vertical directions after moving the relative address pointer in accordance with the motion vector. The motion vector is rounded off to units corresponding to ¼ the SD pixel spacing prior to determining the relative address pointer. However, even in this case, if the x and y components of the motion vector of the SD image deviate significantly from an integral multiple of ¼ of the SD pixel spacing, additional SD pixel data may not be written into the memory. For instance, if the motion vector is determined to be less than ⅛ of the SD pixel spacing, the new high resolution frame may be formed as identical to the previous high resolution frame, and no new SD pixel data is written.

[0139] FIGS. 22-24 further illustrate the enlarged access area approach. FIG. 22 illustrates a portion of an SD pixel layout of pixels P_SD. An object OB_N−4is assumed to be formed by the pixels in frame (N−4). The object's position moves from frame to frame until it reaches the position of object OB_Nin frame N. The pixels forming the image are written in the access area enlarged to the area of 4P_H×4P_Vas described immediately above. The storage states are shown in the access area 15a_i′ of FIG. 23, where the pixels of different frames are shown with different shadings. The motion vectors corresponding to the frame to frame motion are also tabulated in FIG. 23. FIG. 24 is an enlarged view of the memory storage locations or cells MC of FIG. 23. After four frames of motion from frame (N−4) to frame N, many of the storage locations remain empty. These can be filled in by interpolation, by the classification adaptive process, and so forth. As another option, an empty location can be filled by simply inserting the same pixel value of the nearest storage location into the empty location.

[0140] The dashed lines shown in FIGS. 23 and 24 designate alternate rows and columns of pixel data. To read out HD pixel data from access area 15a_i′, it is sufficient to read data of every other row and every other column, i.e., the data from storage locations along the dashed lines. This method will produce higher resolution than that realizable by employing an access area with only a 2P_H×2P_Varray of storage locations.

[0141] Referring to FIG. 27, another embodiment 34′ of resolution converter 34 is shown. This embodiment is similar to that of FIG. 12, except it is designed for the case in which the complete image of each frame moves together by the same amount from frame to frame. That is, resolution converter is functionally similar to distortion corrector 4 a of FIG. 4. Thus, a single high resolution image generator 41 without a switch 22 or buffer memory 23 is used in place of high resolution object generators 411-41M, and area divider 21 and combiner 24 are eliminated. All other aspects are the same as described above for the embodiment of FIG. 12.

[0142] In each of the above-described embodiments of the invention employing a resolution creating memory 15 or 15_i, when previous high resolution frame data was already created in the resolution creating memory, storage of data from the next frame, i.e., from present frame memory 11A, was disclosed to be accomplished by moving the address pointer in accordance with the motion vector, and overwriting some of the prior data. As an alternative to that approach, the motion vector of the previous frame can be stored and compared with the motion vector of the present frame to determine which motion vector has x and y components closest to the distance corresponding to one high resolution pixel interval, and data stored accordingly. In other words, data associated with the motion vector that has a smaller round off error would be selected for storage. For the memory of FIG. 23, this would mean selecting data associated with the motion vector closest to an integer multiple of ¼ of an SD pixel interval.

[0143] While the present invention has been particularly shown and described in conjunction with preferred embodiments thereof, it will be readily appreciated by those of ordinary skill in the art that various changes and modifications may be made to the disclosed embodiments without departing from the spirit and scope of the invention. For example, while the disclosed embodiments were described to reduce aliasing distortion and optionally to generate high resolution images, the invention can also be employed to enlarge images. Further, while the above processes are disclosed as processing data on a frame to frame basis, the processes can be employed to process portions of frames, such as in units of fields. Also, images can be displayed on other displays aside from a CRT such as a liquid crystal display. Further, the received image signal may be a digital image signal. In this case, the A/D converter (e.g., of FIG. 1 or 11 ) can be removed. Moreover, the present invention may be employed to convert an interlaced scanning image to a progressive scanning image. Accordingly, these and other changes and modifications are intended to be included within the scope of the invention as defined by the appended claims.