Embedding and detecting a watermark in images

[0002] This is a continuation-in-part of U.S. patent application Ser. No. 09/226,532, filed Jan. 7, 1999 now U.S. Pat. No. 6,198,832.

[0003] 1. Field of the Invention

[0004] The invention relates to a method and arrangement for embedding a watermark in an image. The invention also relates to a method and arrangement for detecting a watermark in an image. The invention further relates to a method and arrangement for transmitting and receiving an image containing a watermark. Furthermore, the invention relates to a method and arrangement for recording an image containing a watermark onto a storage medium, and a method and arrangement for playing back an image containing a watermark from a storage medium. The expression “image” is understood to mean still images as well as video.

[0005] Digital watermarking is a technique used to certify the ownership of digital image or video contents. By imperceivably hiding such ownership information into digital content, it becomes possible to prevent piracy and illegal use of this content. Typical applications of watermarking include copy-protection of digital video and broadcast monitoring.

[0006] 2. Description of the Related Art

[0007] Most presently used techniques for watermarking digital images boil down to adding a pseudo-noise watermark pattern to an image. This addition may take place in the pixel domain or the frequency domain. Effectively, it means that luminance and/or color components of the watermarked image are slightly different from the original. The detection of a watermark is done by correlating the suspect image with the watermark pattern to be detected. Such a prior art method is disclosed, inter alia, in International Patent Application WO-A-98/03014, corresponding to U.S. Pat. No. 5,933,798.

[0008] A watermarking method must fulfill a number of basic requirements. Adding the watermark pattern may not affect the visual appearance of the image. There should be many different watermark patterns, since every copyright holder should have at least one distinct watermark. Detection of the watermark must be possible without using the original image. The detection must be fast so as to be able to detect watermarks in video, or to scan the World Wide Web for watermarked images. It should not be easy for attackers to remove the watermark. The method must further be robust to transformations or operations that can be applied to images or video, such as, color transformations, geometric transformations, such as, scaling, translation or rotation, data compression, such as, JPEG/MPEG, noise, embedding of other watermarks, gray scaling, image smoothing, dedicated transformations, A/D and D/A conversions, PAL/NTSC conversions, etc.

[0009] The above requirements are conflicting, and there are no methods known that fully satisfy all requirements. Many methods can easily be hacked by attackers who have a number of watermarked images at their disposal. Since detection is based on correlation (which is a linear operation), an attacker needs only a good estimate of the watermark, and he can use this to subtract the guessed watermark from images in order to mislead detection. Simple geometric transformations such as rotations or scaling can only be dealt with at the expense of considerable computing times during detection. It is also difficult to ensure sufficient variation of watermarks. The number of bits that can be embedded using one correlation is limited.

[0010] It is an object of the invention to provide a method and arrangement for embedding a watermark which alleviates the above mentioned drawbacks.

[0011] To this end, the invention provides a method of embedding a watermark in an image, comprising the steps of identifying a given number of most significant image pixels; defining a watermark in the form of a pattern of pixels; and locally warping the image so as to move a significant percentage of the most significant image pixels to the vicinity of the watermark pattern.

[0012] The corresponding method of detecting the watermark comprises the steps of identifying a given number of most significant image pixels; determining the distances of said most significant image pixels to the watermark pattern; and generating a detection signal indicating that said predetermined watermark is embedded in the suspect image, if the average distance of said most significant pixels to the watermark pattern is significantly smaller than the average distance of all image pixels to the watermark pattern.

[0013] The watermark is thus embedded by slightly changing geometric features of the image instead of changing luminance or color components. This makes removal of the watermark a complex task because it requires image dependent processing, in contrast with simply subtracting an estimated watermark from the image, which works for all images in most current methods. The method is also more robust against geometric transformations such as scaling and rotations, because detection can be done in a shorter time. Further, the method allows a sufficient variation of watermark patterns to be used. Checking a suspect image against many different watermark patterns is computationally a drastic improvement over current methods.

[0014] With the above and additional objects and advantages in mind as will hereinafter appear, the subject invention will be described with reference to the accompanying drawings, in which:

[0015] FIG. 1 shows, schematically, an arrangement for embedding a watermark in an image in accordance with the invention;

[0016] FIGS. 2, 3A and 3B show images illustrating the operation of the arrangement shown in FIG. 1;

[0017] FIG. 4 shows, schematically, a first embodiment of an arrangement for detecting a watermark in an image in accordance with the invention;

[0018] FIG. 5 shows a further embodiment of an arrangement for embedding a watermark in an image;

[0019] FIGS. 6A-6B show images of a watermark illustrating the operation of more general arrangements for embedding and detecting a watermark in accordance with the invention;

[0020] FIG. 7 shows, schematically, the more general embodiment of the arrangement for embedding a watermark in an image;

[0021] FIG. 8 shows, schematically, the more general embodiment of the arrangement for detecting a watermark in a suspect image;

[0022] FIG. 9 shows a block diagram of an arrangement for transmitting and receiving an image containing a watermark; and

[0023] FIG. 10 shows a block diagram of an arrangement for recording on a storage medium and for playing back from the storage medium an image containing a watermark.

[0024] FIG. 1 shows an embodiment of an arrangement for embedding a watermark W in an image I in accordance with the invention. The arrangement comprises a salient point extraction module 10, a matching module 11, and a warping module 12. The arrangement receives an input image I and a watermark W.

[0025] FIG. 2 shows an example of the watermark W. It is assumed to be a pattern of lines 20, but this is not essential. A pixel is said to lie “on the watermark”, or “in the vicinity of the watermark”, if the distance from that pixel to the nearest line 20 is less than a threshold value δ. In accordance herewith, pixel 21 lies on the watermark, whereas pixels 22 and 23 do not. In this embodiment, the watermark is dense. This is understood to mean that a significant percentage of the image pixels (for instance, p=50%) lies in the vicinity of one of the lines.

[0026] The salient point extraction module 10 identifies a fixed number K of most significant pixels (hereinafter also referred to as most salient points or salient points for short) of the input image I. Salient points can be, inter alia, the pixels giving the highest response to an edge-detect filter, but there are many other ways of finding significant points. Other examples of significant points are corners of edges, or local maxima or local minima. The saliency of pixels may be derived from the luminance of pixels, the chrominance, or both.

[0027] Visual artifacts may occur in the watermarking method if the significant points are located closely together. To avoid this, the image may be subdivided into blocks, and one or a few significant points per block are chosen unless there is no variation in the image pixels within the block.

[0028] The salient points constitute a set S of image pixels. This set of pixels is applied to the matching module 11, which matches the salient points with the watermark and identifies a set S₁of salient points that lie on the watermark W, and a set S₀of salient points that does not.

[0029] The sets S₀and S₁of salient points are applied to the warping module 12. This module introduces small, local, geometrical changes to the image such that a significant high percentage of pixels from the set S is moved to the vicinity of lines. This is illustrated in FIG. 2 by an arrow 24 which denotes that pixel 22 has been moved to a line of the watermark. The thus warped image I_Wconstitutes the image with embedded watermark. The geometric changes introduced by warping affect neighborhoods of the pixels of S in a smooth way, such that at some distance of a given pixel the geometric change decays to zero. The warping algorithm is preferably such that it does not create too many new significant points. The direction into which a point is warped should be adapted to local image features, such that it does not introduce visual artifacts.

[0030] Warping is well-known in the field of photo processing. FIGS. 3A and 3B have been made with a commercially available PC software package for photo editing. FIG. 3A shows an original image with salient points 31, 32, 33, and watermark pattern lines 34, 35, 36. FIG. 3B shows the watermarked image in which the salient pixels have been warped to the nearest line (for clarity shown in an exaggerated manner).

[0031] In a practical experiment, an image was found to have 1014 significant points, 541 of which (53%) were located in the vicinity of the line pattern. Out of the 473 points not lying on the watermark, 354 points were warped, so that 895 salient points (88%) of the image were located in the vicinity of the pattern after watermarking.

[0032] FIG. 4 shows a first embodiment of an arrangement for detecting the watermark W in an image J in accordance with the invention. The arrangement comprises a salient point extraction module 40 which is identical to module 10 in FIG. 1, a matching module 41 which is identical to module 11 in FIG. 1, and a decision module 42. The arrangement receives a suspect input image J and the watermark W whose presence in the input image J is to be checked. The operation of the arrangement will be obvious in view of the description of the embedder described above. The salient point extraction module 41 determines the set S of most significant pixels in the suspect image by the same method as used in embedding. The matching module 42 matches the salient points with the watermark line pattern. The detection module 43 detects whether a statistically significantly high percentage of the salient points lies within the vicinity of the line pattern (set S₁). If that is the case, the watermark is present and a detection signal D=1 is generated, otherwise it is not (D=0).

[0033] In the embodiment of the embedder described above (in which the watermark pattern covers 50% of the image pixels), the watermark is detected if substantially more than 50% of the salient points of the suspect image lie on the line pattern. In the practical experiment mentioned before, 708 out of 1014 salient points of the watermarked image (70%) were found to lie on the pattern. Note that the number of salient points in the watermarked image can differ from the number of salient points in the original image, because the warping process may affect the “saliency” of pixels.

[0034] FIG. 5 shows a further embodiment of the arrangement for embedding the watermark. The embodiment differs from the arrangement shown in FIG. 1 in that the embedder comprises a detection module 13 which is identical to module 42 in FIG. 4. Thus, the embedder now includes the watermark detector. The detection module 13 applies the detection signal D to the warping circuit 12. Further, the warped image I_Wis fed back to the salient point extraction module 10. The arrangement shown in FIG. 4 embeds the watermark in an iterative sequence of steps. Initially, only a restricted number of salient points of the input image I is warped. The watermarked image I_Wis subsequently fed back to undergo “harder” embedding until the detection module 13 decides that the amount of warping is sufficient. Every time, those pixels are selected for which warping has the least artifacts, for example, the pixels lying nearest to a line. Exaggerated warping of pixels, such as pixel 23 in FIG. 2, can thus be avoided.

[0035] In the embodiments for embedding and detecting a watermark described above, salient pixels are warped such that they lie on the watermark (cf. pixel 22 in FIG. 2). More generally, the closeness to the line pattern is controlled. Instead of simply counting how many points are actually within the vicinity δ of the line pattern, detection can also be based on the average distance between significant points and the line pattern. The corresponding process of embedding is based on changing said average distance.

[0036] The operation of such a more general arrangement will now be described with reference to FIGS. 6A and 6B. FIG. 6A shows the same pattern of lines 20 and salient image pixels 21-23 as shown in FIG. 2. The lines 20 no longer have the “thickness” 2δ. The distance from the salient pixels 21, 22, and 23 to the nearest line is denoted d_i,1, d_i,2, and d_i,3, respectively, where the index i indicates that the distance relates to pixels of the input image. FIG. 6B shows the pattern and image pixels after watermarking. The salient pixels 21-23 have been warped such that their respective distances to the nearest line 20 are changed (decreased or increased). Not all pixels need to be warped. Neither need the pixels to be warped by the same distance or factor. In FIG. 6B, the distances after warping are denoted d_w,1, d_w,2, and d_w,3, respectively.

[0037] Before describing the operation of an embedder and detector in accordance with this embodiment, it is noted that the average distance {overscore (d)}₁of the salient points (the set S) of an unwatermarked image, i.e., $\begin{array}{cc}{\stackrel{\_}{d}}_i=\frac{1}{K}\sum _{k\in S}{d}_{i,k}& \left(\mathrm{Eq}.1\right)\end{array}$

[0038] (where K is the number of salient pixels), is equal to the average distance of all pixels of the image, provided that the number of salient points is large enough and uniformly distributed. In accordance herewith, the average distance {overscore (d)}_iis statistically independent of the image content and depends on the watermark pattern W only.

[0039] FIG. 7 shows, schematically, the corresponding embodiment of the arrangement for embedding the watermark W in an image I. The arrangement comprises a salient point extraction module 70 and a warping module 71. They are identical to modules 10 and 12, respectively in FIG. 1.

[0040] FIG. 8 shows, schematically, the corresponding embodiment of the arrangement for detecting the watermark W in a suspect image J. The arrangement comprises a salient point extraction module 80, a circuit 81 for determining the average distance {overscore (d)}_wof the salient points of the possibly watermarked image J in accordance with ${\stackrel{\_}{d}}_w=\frac{1}{K}\sum _{k\in S}{d}_{w,k},$

[0041] a circuit 82 for determining the average distance {overscore (d)}₁of the salient points of the unwatermarked image in accordance with Eq. 1 above, and a detection circuit 83. It will be appreciated that the detection circuit 83 concludes that the suspect image J is watermarked (D=1) if the average distance {overscore (d)}_wis significantly smaller than the average distance {overscore (d)}_i, otherwise it is not (D=0).

[0042] The watermarking method in accordance with the invention has low visual impact. Since the line pattern is dense with respect to the image, one can always find sufficiently many candidates of significant pixels that can be warped without affecting the image too much. Dedicated significant point location and sophisticated warping further help to reduce visual artifacts.

[0043] The invention further supports sufficient variation of watermarks by having a large library of line patterns to look for. Not only can it easily be verified whether one given watermark is embedded, but due to its relative computational simplicity, it is possible to check for many different line patterns.

[0044] As regards speed of detection, a complexity analysis of the detection method will now be given. For each watermark to be tested, the K most significant points of an image with resolution n²are determined. This operation requires O(n²) time. Once these K points of an image have been selected, matching them with many different watermark line patterns is computationally a considerably simpler task than computing correlations between the complete image and equally many watermark patterns as taught by the prior art. Namely, for an image of size n*n pixels, computing a correlation requires O(n²) time. Note that a dense line pattern contains O(n) lines, and finding the closest line to a given query point requires O(log n), when preprocessing of the line pattern is allowed. Once the K points have been found, the time complexity per tested watermark becomes O(k log n).

[0045] The invention further provides embedding more bits. Different line patterns can be used in different areas of the picture. Different line patterns (each of lower density) can be combined. It is also possible to apply different significant point extraction methods. For example, one method is used for extracting local maxima only, one method is used for extracting upper-right corners only, etc.

[0046] The embedded watermark is hard to remove by an attacker. It is not easy for an attacker to determine the specific line pattern that forms the watermark. Even when an attacker has a good guess of the pattern, it is still not straightforward to remove the watermark because the attacker will have to apply warping as well, which is more complicated than simply subtracting a noise pattern.

[0047] The method is robust because the significant point extraction can be implemented such that transformations that sufficiently preserve the quality or the essentials of an image (such as compression, noise addition, or color changes) will leave the locations of the most significant pixels intact. With regards to scaling and geometric transformations, it should be noted that matching scaled or rotated sets of points with scaled or rotated line patterns is a computationally easier task than computing correlations.

[0048] It is to be noted that the pattern onto which significant pixels are warped need not necessarily be a line pattern and its vicinity. Arbitrary dense point patterns may also be used.

[0049] Thus, a digital image signal is watermarked by locally changing geometric features of the image. The watermark consists of a pseudo-random, dense subset of image pixels, e.g., a pattern of lines (20). A number of significant image pixels (21,22,23), i.e., pixels which give the highest response to a predetermined processing operation, is determined and then moved (24) to the vicinity (δ) of the line pattern. As a result of this “warping”, the majority of significant image pixels (21,22) is eventually located within the vicinity of the line pattern. At the receiver end, the most significant pixels of an input image are again determined. The image is a watermarked image if a statistically high percentage lies within the vicinity of the line pattern.

[0050] FIG. 9 shows an arrangement for transmitting and receiving an image containing a watermark. A watermark embedder 100 receives and image signal I on input 102 and a watermark W on input 104. The watermark embedder 100 is constructed in accordance with any of the embodiments of FIG. 1, 5 or 7. The image containing the watermark I_Wis supplied at output 106 to a transmitter 108. The transmitter 108 then converts the watermarked image I_Winto a transmission signal T_Sand applies transmission signal T_Sto a transmission medium 110. The transmitter 108 and the transmission medium may take any known form, for example, in the case of a broadcast, the transmitter 108 modulates the watermarked image I_Wonto a radio frequency (RF) carrier wave, and the transmission medium 110 may comprise airwaves containing the radio waves, or a cable carrying the radio waves. Alternatively, it is known to supply digital signals via the Internet. As such, the transmitter 108 may comprises means for transmitting the watermarked image I_Wover the transmission medium 110 comprising the Internet.

[0051] A receiver 112 coupled to the transmission medium 110 then receives the transmission signal T_Sand applies the image signal J to one input of an AND-gate 114. A watermark detector 116 is connected to also receive the image signal J from the receiver 112 as well as the watermark W. If a watermark is not detected, the watermark detector 116 applies a logic “1” signal to the AND-gate 114 which allows the image signal to be applied to a display device 116. If the watermark is detected and if the user is not authorized to receive the image signal, the watermark detector 116 sends a logic “0” signal to the AND-gate 114 which blocks the application of the image signal J to the display device 118. Finally, if the user is authorized, the watermark detector 116 applies the logic “1” signal to the AND-gate 114 which, again, applies the image signal J to the display device 118.

[0052] FIG. 10 shows a block diagram of an arrangement for recording and playing back a watermarked image signal IW onto and from a record carrier. This arrangement is substantially similar to that shown in FIG. 9, except that the watermarked image signal IW at the output 106 of the watermark embedder 100 is applied to a recording device 120. The recording device 120 then converts the watermarked image signal IW into a recording signal RS and records the recording signal RS onto the record carrier 122. The recording device 120 and the record carrier 122 may take any known form. For example, in the case of the record carrier 122 being a magnetic video tape, the recording device 120 may take the form of a helical scan video tape recorder. Alternatively, the record carrier 122 may be an optical disk, for example, CD-ROM, CD-R, DVD, DVD-ROM, DVD-R/W, etc. In that event, the recording device 120 would take the form of an optical disk recorder.

[0053] In order to play back the recording signal RS, the record carrier 122 is inserted into a playback device 124 which processes the recording signal RS and generates an image signal J. As with the embodiment of FIG. 9, the playback device 124 applies the image signal J to one input of an AND-gate 114. A watermark detector 116 is connected to also receive the image signal J from the playback device 124 as well as the watermark W. If a watermark is not detected, the watermark detector 116 applies a logic “1” signal to the AND-gate 114 which allows the image signal J to be applied to a display device 116. If the watermark is detected and if the user is not authorized to playback the image signal, the watermark detector 116 sends a logic “0” signal to the AND-gate 114 which blocks the application of the image signal J to the display device 118. Finally, if the user is authorized, the watermark detector 116 applies the logic “1” signal to the AND-gate 114 which, again, applies the image signal J to the display device 118.

[0054] Numerous alterations and modifications of the structure herein disclosed will present themselves to those skilled in the art. However, it is to be understood that the above described embodiment is for purposes of illustration only and not to be construed as a limitation of the invention. All such modifications which do not depart from the spirit of the invention are intended to be included within the scope of the appended claims.