MPEG2 moving picture encoding/decoding system

[0002] 1. Field of the Invention

[0003] The present invention can be used for a Moving Picture Experts Group (MPEG) moving picture compression/decompression application of motion prediction/compensation based on a discrete cosine transform and minimizes degradation of image quality.

[0004] This invention relates to a digital copy protection application of moving picture data which is seldom removed intentionally or unintentionally by a user except an author, and more particularly, relates to an MPEG2 moving picture encoder/decoder.

[0005] 2. Description of the Related Art

[0006] The MPEG2 standard is a compression/decompression standard for video applications, and exploits temporal redundancy for motion compensated interpolated and predicted encoding. That is, the assumption is made that “locally” the current picture can be modeled as a translation of the picture at a previous and/or future time. “Locally” means that the amplitude and direction of the displacement are not the same everywhere in the picture.

[0007] The MPEG2 standard specifies predicted and interpolated interframe encoding and spatial domain intraframe encoding. It has block based motion compensation for the reduction of temporal redundancy, and a block based Discrete Cosine Transform based on compression for the reduction of spatial redundancy. The information relative to motion is based on a 16×16 array of pixels and is transmitted with the spatial information. Motion information is compressed with variable length codes, such as Huffman codes.

[0008] Recently, audio/video information expressed as digital information is becoming more widely used and a method utilizing digital products also has been increasing accordingly as digitalization of the A/V application and popularity of the Internet have been increasing rapidly.

[0009] Specifically anybody who is able to use a PC can copy/edit digital products easily and accordingly a social issue of illegal copying has been raised. A watermark technique has become prominent as a solution to prevent this problem.

[0010] There are two exemplary methods for providing copy protection of digital A/V data to prevent illegal copying. The first method is encryption, i.e., a copy protection method by scrambling the digital information. The second method is a digital watermark method with the purpose of preventing the illegal use of digital information.

[0011] The first method is a technique for prevention of illegal copying of digital A/V data, by providing descramble information and password information capable of accessing and running the A/V product only in the case that the A/V product is bought legally.

[0012] The second method is a technique which utilizes self-restraint by a user to not produce an illegal copy of the A/V product by embedding ID information or a logo in a state of noise into A/V contents data of the A/V product for the purpose of forbidding the illegal or commercial use of digital information. The watermark technique is used on the original image and is invisible to a person who would copy it, but the author can prove that the copied image is his by virtue of an arbitrary reverse processing.

[0013] For example, in a case where a counterfeiter forges money using a color copier, a vignette on the original bill turns clear by copying the bill, and consequently, it makes it virtually impossible to copy a bank note. This is called a visible watermark.

[0014] Also, in the case where a spy writes a message onto paper with salt water, other people think this is ordinary paper, but this paper is a medium having important information for the spy. The spy can see the message anytime he wants to see by heating the paper. In this case, we call it an invisible watermark.

[0015] At present, the watermark technique is used for digital still images or audio, i.e., putting the message distinguishable from the original image into the image. Therefore, in case that an author's own image circulates illegally, the image can be proved to be that of author by performing an arbitrary reverse processing.

[0016] Thus, techniques for preventing the illegal copying of digital product are increasingly being studied these days.

[0017] FIGS. 1 and 2 illustrate a conventional MPEG2 moving picture encoder and decoder respectively. FIG. 3 illustrates a structure of a video picture used in the MPEG2 moving picture encoder/decoder and, FIG. 4 illustrates three types of pictures and their relationship under the MPEG2 standard. We will explain the conventional MPEG2 moving picture encoder and decoder by referring to these figures.

[0018] MPEG encoding is a hybrid type lossy coding technique wherein the redundancy information which the image signals have in the spatial domain and the temporal domain are removed and the data are compressed (refer to FIG. 3). At this time, the compression technique of spatial domain is called intra-coding and the image data used in intra-coding are called intra-picture (in short, I picture). The compression technique of temporal domain is called inter-coding and in this case, the image data are classified by two according to two prediction methods. The first one is a predicted picture (P picture) wherein prediction error rate of a forward direction is encoded, and the second one is an interpolated picture or bi-directional picture (B picture) wherein a prediction error rate of the bi-directional direction is encoded.

[0019] In other words, the I picture is encoded independently of other near pictures (in this instance, the picture is a frame signal or a field signal). In the P picture, the difference signals of predicted/interpolated movement are encoded only after considering the correlation of the movement of the previous I picture or P picture. In the B picture, the difference signals of predicted/interpolated movement are coded only after considering the correlation of the movement of the previous I or P picture and the next I or P picture.

[0020] Among the three modes, that is, the forward direction mode, backward direction mode, and forward and backward direction mode, the mode having the smallest value of prediction error rate is selected in the prediction/interpolation method of the B picture.

[0021] The picture structure of the MPEG recommendation (called a main profile, main level; MP@ML) is I, B, B, P, B, B, P . . . pictures and needs a frame memory 110 which can store at least three pictures (the cycle of a picture).

[0022] A field/frame adaptive coding method is possible in an MPEG2 encoding method so as to increase coding efficiency. The unit of the picture can be defined as a field or frame according to the purpose of the encoding. A frame/field memory 112 stores the field data or frame data to be coded.

[0023] A subtractor 134 receives the field data or frame data from the frame/field memory 112 and interpolated predicted motion data from an adaptive estimator 130, to perform a subtraction of local decoded I or P pictures, wherein the motion prediction is interpolated to encode the prediction error rate of the P and B pictures, and the pictures which are now input. A Discrete Cosine Transformn (DCT) 114 performs an orthogonal transform which transforms image signals from the subtractor 134 spatially structured into image signals of the frequency domain. A quantizer 116 approximates signals to a typical value to map the DCT-transformed image signals to a code book which is defined in a variable length coder (VLC). Data loss occurs in the quantizer 116.

[0024] A dequantizer 122 performs an inverse process of the quantizer 116 for encoding the prediction error rate of P and B pictures. An inverse DCT (IDCT) 124 performs an inverse process of the DCT 114 for encoding the prediction error rate of the P and B pictures. A subtractor 126 performs a subtraction operation on the output from the IDCT 124 and the interpolated predicted output from the adaptive estimator 130. A frame memory 128 stores local decoding images output from the subtractor 126 according to the dequantizer 122 and the IDCT 124.

[0025] A motion estimator 132 encodes the prediction error rate of the P and B pictures output from the frame/field memory 112 and the adaptive estimator 130 is a motion compensator which interpolates predicted motion, providing its output to the subtractors 126 and 134.

[0026] An activity calculator 118 reflects the characteristics of the complexity of the input images to the quantizer 116 and a rate controller 120 sets up the quantizer 116 so that an overflow/underflow of an output buffer 138 does not happen. A VLC & MUX (variable length coder and multiplexer) 136 entropy encodes and multiplexes the signals output from the rate controller 120, the quantizer 116 and the motion estimator 130. The output buffer 138 provides a buffer for the MPEG coded bit stream output from the VLC & MUX 136.

[0027] FIG. 2 shows the conventional MPEG2 moving picture decoder. A buffer 150 stores the coded bit stream. A VLD (variable length decoder) & DEMUX 152 performs an inverse process of the VLC & MUX 136 (of FIG. 1) to decode the MPEG coded bit stream. A dequantizer 154 dequantizes the coded MPEG coded bit stream output from the VLD & DEMUX 152, and an IDCT 156 performs an inverse process of DCT of the output of the dequantizer 154. Adder 160 adds the output of the IDCT 156 with an output of a multiplexer (MUX) 170. A previous picture store 162 is a memory for motion compensation of the P or B picture output from the adder 160. A future picture store 164 is a memory for motion compensation of the P picture output from the adder 160. An adder 166 performs an addition of the outputs of the previous picture store 162 and the future picture store 164 when the motion prediction of the B picture is bi-directional. A ½ multiplier 168 multiplies the interpolation of average values when the motion prediction of the B picture is bi-directional, and the MUX 170 multiplexes the outputs of the previous picture store 162, the ½ multiplier 168, the future picture storer and a “0” bit.

[0028] One picture can be divided into uniformly sized regular square areas and each area is transformed. Therefore, the image is divided into image ingredients of different frequencies from an average value (DC value) to an image ingredient value of an extremely high frequency. This division process is called an orthogonal transformation and the orthogonal transformation is a discrete cosine transform (DCT).

[0029] Orthogonal transformations, because they have a frequency domain interpretation, are filter bank oriented. This means that the purpose of the DCT is to reduce the correlation of the image information. Since each DCT-transformed coefficient indicates individual frequency information, the correlation of adjacent coefficients is rare. The discrete cosine transform is also localized. That is, an encoding process illustrates samples on an 8×8 spatial window which is sufficient to compute 64 transform coefficients or sub-bands.

[0030] Another advantage of the discrete cosine transform is that fast encoding and decoding algorithms are available. Additionally, the sub-band decomposition of the discrete cosine transform is sufficiently well behaved to allow effective use of psychovisual criteria.

[0031] After the discrete cosine transform, many of the higher frequency coefficients are zero. These coefficients are organized into a zigzag, as illustrated in FIG. 5, and converted into run-amplitude (run-level) pairs. Each pair indicates the number of zero coefficients. This is coded in a variable length code.

[0032] Discrete cosine transform encoding is carried out in the three stages as illustrated in FIG. 5. The first stage is the computation of the discrete cosine transform coefficients. The second stage is the quantization of the coefficients. The third stage is the conversion of the quantized transformation coefficients into run-amplitude pairs after reorganization of the data into a zigzag scanning order.

[0033] Quantization can be viewed as a shift to the right by several bits. Quantization enables a very high degree of compression, and a high output bit rate, and retains high picture quality. Quantization can be adaptive with an I picture having fine quantization to avoid “blockiness” in the reconstructed image. This is important because I pictures contain energy at all frequencies. By way of contrast, P and B pictures contain predominately high frequency energy and can be coded at a coarser quantization.

[0034] One challenge facing decoder designers is the accommodation of a single decoder system to a variety of display output formats, while complying fully with luminance/chrominance relationships and the MPEG2 standard. The displayed output of the decoder chip must conform to Consultative Committee International Radio (CCIR) recommendation 601. This specifies the number of luminance and chrominance pixels in a single active line, and also how the chrominance pixels are subsampled relative to the luminance signals.

[0035] The format defined as 4:2:2 is supported in most cases in industry. This defines 720 active luminance signals, and 360 color differentiated signals, where each line of luminance signals has a corresponding line of chrominance signals. CCIR recommendation 656 goes on to define the number of active lines for National Television System Committee (NTSC) and Phase Alternation by Line (PAL) environments as 480 and 576, respectively. The contents as noted above are disclosed in U.S. Pat. No. 5,668,599.

[0036] The MPEG2 moving picture encoder 100 performs an encoding method by utilizing the cooperation of an intracoding method on the spatial domain and an interceding method on the temporal domain. The MPEG2 moving picture encoder 100 performs the intracoding method on the spatial domain by compressing the original image into a variable length coding of a Huffman code through the DCT 114 and the quantizer 116 and transmits the variable length code.

[0037] The MPEG2 moving picture encoder 100 performs the intercoding method on the temporal domain by decompressing the I picture compressed on the spatial domain through the dequantizer 122 and the Inverse Discrete Cosine Transform (IDCT) 124, and predicts by comparing the compressed I picture with the image being input at present through the frame memory 128 and the adaptation estimator 130, and then encodes a difference signal with the original signal by compensating motion, i.e., spatial-shifting the image being input at present as much as the predicted motion.

[0038] In the case that a method predicting motion is forward prediction, we call it the P picture and in case that a method predicting motion contains all of forward and backward predictions, we call it the B picture. Accordingly, motion prediction and compensation of P and B images are affected by the picture accuracy coded as the I picture. So, in the decoding process decoding the encoded image, first, the I picture must be decoded exactly so that the P and B images, to which the difference signals are transmitted, can be decoded accurately.

[0039] But even through the use of the copy preventing technique by encryption and scrambling as noted above, it is possible to easily copy data when duplication and key data are known. And also the watermark technique for moving picture data has some problems which can reduce encoding efficiency by embedding ID information and a logo in the form of noise.

[0040] The picture structure of an MPEG2 moving picture encoding method, as illustrated in FIG. 4, includes an intraframe (I picture) reducing spatial redundancy information of image information, a predicted frame (P picture) reducing interrelation between frames through forward prediction, and an interpolated frame (B picture) reducing between frames through bi-direction prediction.

[0041] Therefore, in the decoding of the image signal, only in the case where the decoded previous I picture exists, can the P picture be decoded perfectly through motion compensation, and only in the case where the decoded I and P pictures are used in B picture prediction in the encoding process, can the B picture be decoded through motion decoding.

[0042] Up to now, digital watermark information discrete-cosine-transformed in the form of noise is embedded into an original image and an I picture codes this digital watermark information.

[0043] And the case of predicting motion of the P and B pictures is performed according to encoding locally the I picture. Consequently there is a problem of an error while estimating motion of the P and B pictures by the mixed watermark information.

[0044] Because the watermark technique for still images, which has started to be studied recently as noted above, includes watermark data on the spatial domain, the watermark technique is not suitable for an MPEG encoding method compressing data by removing redundancy information in accordance with using interrelation of data on the spatial domain and the temporal domain.

[0045] That is, in case the image quality of the image that contains the watermark information deteriorates conspicuously in comparison with the image quality where the watermark information is not contained, the meaning can be lost because the image can deteriorate even though the original object contains watermark information.

[0046] Thus, the image containing the watermark information has to appear very similar to the image which does not contain the watermark information.

[0047] The present invention is devised to solve the above-noted problems and can be used by MPEG moving picture compression/decompression applications of motion prediction/compensation based on the discrete cosine transform.

[0048] Accordingly, it is a first object of the invention to provide a digital copy protection apparatus of moving picture data that makes it difficult to get rid of watermark information by a user except an author, in accordance with minimizing degradation of pixels.

[0049] A second object of the present invention provides an MPEG2 moving picture encoder and a third object of the present invention provides an MPEG2 moving picture decoder.

[0050] Also, a fourth object provides an MPEG2 moving picture encoding/decoding system with the encoder and the decoder.

[0051] Additional objects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

[0052] According to a first aspect of the present invention to achieve the second object, an MPEG2 moving picture encoder includes a frame memory, a frame/field memory, an activity calculator, a rate controller, a Discrete Cosine Transform (DCT) unit, a quantizer, a dequantizer, an Inverse Discrete Cosine Transform (IDCT) unit, an adaptation predictor, a motion predictor, a variable length Coding & Multiplexer VLC & MUX and a buffer, and further includes a digital watermark inserter to generate and discrete-cosine-transform a digital watermark on a frequency domain and is installed between the DCT and the VLC & MUX; and a digital watermark remover to remove digital watermark information on a spatial domain of the DCT embedded for preventing an error while estimating motion on a temporal domain for P and B pictures by being installed between the dequantizer and the IDCT.

[0053] In a second aspect of the present invention to achieve the third object, an MPEG2 moving picture decoder includes a first buffer to receive and save an encoding bit stream; a variable length decoder and demultiplexer (VLD & DEMUX) to receive the encoding bit stream from the first buffer and to various-length-code, and then output the various-length-coded bit stream; a dequantizer to receive and dequantize the various-length-coded bit stream output from the VLD & DEMUX; a digital watermark remover to remove the digital watermark which has been embedded, to locally decode by reading watermark information and information from the dequantizer; a first inverse-discrete cosine-transform IDCT to inverse-discrete-cosine-transform the signal from the digital watermark remover; a digital watermark inserter to again embed the digital watermark removed by the digital watermark remover for performing local decoding, for the purpose of the decoding output; a second IDCT to inverse-discrete-cosine-transform the information from the digital watermark inserter; a second buffer, when the video signal decoded from the second IDCT and the motion compensation data are added and then the picture in which the digital watermark is embedded is replayed, saving temporarily and outputting it; a multiplexer (MUX); a previous picture store block in which the first IDCT output and the MUX output are added and saved temporarily; and a future picture store block in which the first IDCT output and the MUX output are added and saved temporarily, to transmit the saved value into the previous picture store block.

[0054] It is preferable that the MUX receives the signal from the previous picture storing block, and receives the signal from the future picture storing block, and receives the signal 0, and receives the signal which is generated after the signals from the two blocks are added and ½ multiplied (treated).

[0055] And according to a third aspect of the present invention to achieve the fourth object, in an MPEG2 moving picture encoding/decoding system, includes an MPEG2 moving picture encoder, to output an encoded bit stream after inputting and encoding a video input signal, the MPEG moving picture encoder including copy protection means using a watermark for MPEG2 moving picture data; and an MPEG2 moving picture decoder, to decode the encoded bit stream from the MPEG2 moving picture encoder to output a decoded video signal, the decoder including means for replaying perfectly the image into which the watermark is embedded.

[0056] These and other objects and advantages of the invention will become apparent and more readily appreciated from the following description of the preferred embodiments, taken in conjunction with the accompanying drawings of which:

[0057] FIG. 1 shows a conventional MPEG2 moving picture encoder.

[0058] FIG. 2 shows a conventional MPEG2 moving picture decoder.

[0059] FIG. 3 shows a structure of a video picture used in a conventional MPEG2 moving picture encoder/decoder.

[0060] FIG. 4 shows three types of pictures and their relationship under the MPEG2 standard, i.e., I pictures, P predicted pictures, and B bi-directionally predicted pictures.

[0061] FIG. 5 shows a sequence wherein discrete cosine transform coefficients are calculated and mixed and the discrete cosine transform coefficients are quantized in a zigzag manner.

[0062] FIG. 6 shows a block diagram of an MPEG2 moving picture encoder according to an embodiment of the present invention.

[0063] FIG. 7 shows a block diagram of an MPEG2 moving picture decoder according to the embodiment of the present invention.

[0064] FIG. 8 shows a block diagram of a Discrete Cosine Transform unit, digital watermark inserter and a quantizer shown in FIG. 6.

[0065] While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that it is not intended to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims.

[0066] FIG. 6 illustrates a block diagram of an MPEG2 moving picture encoder 200 according to an embodiment of the present invention and FIG. 7 illustrates a block diagram of an MPEG2 moving picture decoder 245 according to the embodiment of the present invention

[0067] In FIG. 6, the MPEG2 moving picture encoder includes a frame memory 210, a frame/field memory 212, a subtractor 234, a DCT unit 214, a digital watermark inserter 240, a quantizer 216, an activity calculator 218, a rate controller 220, a dequantizer 222, a digital watermark remover 242, an IDCT 224, an adder 226, a frame memory 228, an adaptive estimator 230, a motion estimator 232, a VLC & MUX 236, and a buffer 238. The frame memory 210, frame/field memory 212, subtractor 234, DCT 214, quantizer 216, activity calculator 218, rate controller 220, dequantizer 222, IDCT 224, adder 226, frame memory 228, adaptation estimator 230, motion estimator 232, VLC & MUX 236 and buffer 238 operate in the same fashion as the frame memory 110, frame/field memory 112, DCT 114, quantizer 116, activity calculator 118, rate controller 120, dequantizer 122, IDCT 124, adder 126, frame memory 128, adaptation estimator 130, motion estimator 132, VLC & MUX 136 and buffer 138, respectively, shown in FIG. 1 and previously described.

[0068] The digital watermark inserter 240 makes digital watermark information not removable by a user except an author and is embedded on a frequency domain 8×8 discrete-cosine-transform of the encoded video signal, in order to reduce the burden on hardware. That is, the process of the digital watermark insertion by the digital watermark inserter 240 embeds the watermark information on the frequency domain, so that a duplicator cannot edit or remove a digital product illegally and arbitrarily.

[0069] Referring to FIG. 6, the digital watermark inserter 240 is located between the Discrete Cosine Transform (DCT) unit 214 and the variable-length coder & multiplexer (VLC and a MUX) 236 and is also located between the DCT unit 214 and the quantizer 216. It is preferable that the digital watermark inserter 240 is located immediately between the DCT 214 and the quantizer 216 for visual effect.

[0070] Because basically the MPEG2 standard embodies a frequency domain processing on the discrete cosine transform, the digital watermark inserter 240 of FIG. 6 discrete-cosine-transforms the digital watermark information, and then embeds the digital watermark information. The watermark information is processed and embedded by in the digital watermark inserter 240 and then is added to an original image. In particular, the digital watermark inserter 240 includes a watermark pattern generator 284 to generate digital watermark information (pattern), and a DCT 282 to discrete-cosine-transform the digital watermark information. The digital watermark inserter 240 further includes an adder 280 which adds the discrete-cosine-transformed digital watermark information from the DCT 282 to the output of the DCT unit 214. The added result from the added 280 is input to the quantizer 216.

[0071] Thus, the watermark information embedded on the frequency domain is quantized in the quantizer 216 and is variable-length-coded in the VLC & MUX 236. This is an I picture coding process.

[0072] The dequantizer 222 dequantizes the I or P-picture coded data in which the watermark information is embedded and a digital watermark remover 242 gets rid of the embedded watermark information. An IDCT unit 224 reverse-discrete-cosine-transforms the I or P-picture coded data for estimating correctly the motion information. Therefore, the MPEG2 moving picture encoder 200 prevents an error of a temporal domain coding by the watermark information. The MPEG2 moving picture encoder 200 compensates for the error by estimating an image motion that is being input at present from the I or P picture restored like this, and then codes a difference signal.

[0073] In this instance, the watermark information is embedded on the spatial domain in the back of the DCT 214. The digital watermark inserter 240 embeds the digital watermark information only on the spatial domain of the discrete cosine transform (on the spatial domain of its frame).

[0074] The digital watermark remover 242 removes the digital watermark information on the spatial domain of the discrete cosine transform which was embedded for preventing the error during the motion estimation on the temporal domain for P and B pictures by using the embedded digital watermark information.

[0075] FIG. 7 illustrates the MPEG2 moving picture decoder 245 to correctly decode an MPEG2 video stream including the digital watermark information embedded as in the MPEG2 moving picture encoder 200 shown in FIG. 6.

[0076] The MPEG2 moving picture encoder includes a first buffer 250, a VLD & DEMUX 252, a dequantizer 254, a digital watermark remover 270, a first DCT 256, an adder 260, a previous picture store block 262, future picture store block 264, an adder 266, a ½ multiplier 268, a digital order mark inserter 272, second IDCT 274, a second buffer 258, and an adder 276. The buffer 250, VLD & DEMUX 252, dequantizer 254, first IDCT 256, second buffer 258, adder 260, previous picture store block 262, future picture store block 264, adder 266, ½ multiplier 268, operate in the same function as the first buffer 150, VLD & DEMUX 152, dequantizer 154, IDCT 156, adder 160, previous picture store block 162, future picture store block 164, adder 166, and ½ adder 168, respectively, shown in FIG. 2 and described above. The digital watermark remover 270 removes the watermark information embedded on the spatial domain of the discrete cosine transform while restoring motion of the P and B pictures.

[0077] The MPEG2 moving picture decoder 245 performs operations in the reverse order of the MPEG2 moving picture encoder 200.

[0078] Accordingly in the MPEG2 moving picture decoder 245, the watermark information embedded in the MPEG2 moving picture encoder 200 has to be transmitted.

[0079] In the case that an encoded MPEG video stream is transmitted through a storage medium such as a disk, it is possible to perfectly restore motion, referring to it while decoding by embedding the watermark on a lead-in domain which the user cannot access directly.

[0080] The digital watermark apparatus according to the present invention for preventing the illegal copying of MPEG2 moving picture data makes it possible to prevent degradation of the image quality and efficiency reduction of encoding by the embedded digital watermark information by decreasing the error while estimating motion of the P and B pictures by using the embedded digital watermark information.

[0081] In case of decoding locally for compensating motion, once the embedded watermark information is removed while encoding the I or P picture, and the embedded watermark information is decoded locally, then the embedded watermark information is used as data for motion compensation while decoding the P and B pictures.

[0082] And in case of decoding for the decoding output, the MPEG2 moving picture decoder 221 again embeds the watermark information that had been removed once for decoding locally in the image signals, and reverse-discrete-cosine-transforms the image signals to which the watermark information is embedded, and then adds motion compensation data thereto.

[0083] Conclusively, the image in which the watermark information is embedded is replayed perfectly. Further, there is an effect of decreasing degradation of the image quality to a maximum extent by restraining an increase of encoding information according to the organization of the present invention as noted above, and that the watermark information is not removed easily by a user, except an author.