System for embedding additional information in audio data

[0002] The present invention relates to a method and a system for embedding additional information, such as copyright information, in digital audio data, and for detecting the embedded information. In particular, the present invention pertains to a method and a system for precisely detecting additional information that has been embedded when the audio data in which the additional information has been embedded are so transformed that the sonic quality of the audio data is not drastically deteriorated.

[0003] At present, digital music is provided not only on CDs but also across the Internet. Such digital audio data are so stable that regardless of how often the data are played, the sonic quality is not deteriorated. However, since large quantities of the data can be copied easily, techniques for preventing the illegal copying of data have become ever more important. To prevent such illegal copying, copyright information can be embedded in audio data to clearly evidence the existence of a copyright, or information concerning a distribution destination can be embedded so that the route along which an illegal copy travels can be traced. In order not to reduce the value of the audio components, changes in sonic quality due to the embedding of information must not be aurally perceivable by humans. Furthermore, since processing such as filtering, the compression/decompression performed with MPEG, AC3 or ATRAC, digital to analog and analog to digital conversions, trimming, and the changing of the replay speed may be performed using the digital components, the embedded information must survive even through the changing, the loss, the insertion or the re-sampling of data values that occurs in a range wherein the sonic quality of audio data is not drastically deteriorated.

[0004] The conventional detection method for embedding additional information in audio data is superior in the maintenance of secrecy. However, when additional information is embedded with less modification so that it is imperceptible to human beings, the embedded information could be lost during data processing activities, such as compression/decompression, filtering and digital to analog conversion, so that the conventional method has a problem as for robustness. The ordinary method used for embedding additional information in audio data is a spread spectrum method that uses PN (Pseudo-random Noise) modulation. According to this method, in the time domain, additional information is modulated using pseudo-random noise and the resultant information is embedded. As a result, in the frequency domain, the spectrum of a component that corresponds to the embedded information appears to spread. This method is disclosed in U.S. Pat. Nos. 4,979,210, 5,073,925 and 5,319,735.

[0005] According to the above methods, bit data Bm is modulated using pseudo-random numbers Rn consisting of +1 and 1 generated by a suitable encryption technique (e.g., DES), and the resultant bit information is embedded in audio samples An as follows.

[0006] A′_Nm+n=A_Nm+n+_cB_mR_Nm+n  Expression 1

[0007] wherein Bm denotes +1 or 1 that represents one bit, n=0, 1, . . . , N=1, and c denotes the strength for embedding information. For the detection of the embedded information, bit information is detected by calculating $\begin{array}{cc}{B}_m=\frac{1}{{\hspace{0.17em}}_{c}N}\sum _{n=0}^{N-1}{A}_{\mathrm{Nm}+n}^{\prime }{R}_{\mathrm{Nm}+n}& \text{Expression 2}\end{array}$

[0008] This is because it can be expected that if sequence Rn is random, the values ΣA_Nm+nR_Mn+nwill cancel each other out. To embed information at time domain, the perception frequency response of a human being can not be utilized and the deterioration of the sonic quality can not be prevented. Thus, if information is embedded with small modification level so that human beings can not aurally perceive the modification in the sonic quality due to the embedding, the embedded information will not survive the execution of some post-processes, such as compression/decompression.

[0009] On the other hand, according to the technique in U.S. Pat. No. 5,687,191, in the embedding information process, samples constituting time signals are divided by a polyphase filter to obtain frequency bands, and in each frequency band, information is modulated with pseudo-random noise and the information is embedded. The advantages conferred by this are that different embedding strengths can be employed for the individual frequency bands, and that the frequency property of a human being's aural perception can be utilized. Therefore, with this method, unlike with the other conventional methods, information can be embedded that can be robust and that does not deteriorate the sonic quality.

[0010] According to the methods disclosed in U.S. Pat. Nos. 5,613,004 and 5,687,236, as well as in the present invention, information is embedded in the frequency component that is transformed, and thereafter is detected. With these methods, in order to improve secrecy, the embedding and detection means in a frequency domain is proposed as a signal spreading means. However, in these USPs, the embedding and detection method that has a high sonic quality and is robust is not proposed. With these methods, uncompressed digital audio samples are divided into areas that do not overlap each other (called windows), and an FFT (Fast Fourier Transform) is applied for each of the individual windows. A primary mask and a convolutional mask are employed to determine whether a one bit signal should be embedded in a frequency component obtained by means of the FFT. The primary mask and the convolutional mask are pseudo-random bits, and the size of the primary mask corresponds to the value of the frequency. Each window corresponds to a specific position in the convolutional mask. Whether information is to be embedded in the frequency components of the windows is determined depending on whether the result of the logical calculation of the bit value of the primary mask, at the position corresponding to the frequency, and the bit value of the convolutional mask, at the position corresponding to the window, is true or false.

[0011] According to the embedding method disclosed in U.S. Pat. No. 5,613,004, map information bits (redundant bits produced from additional information) are embedded in specific bit positions of the embedding frequency component. In U.S. Pat. No. 5,687,236, bits are embedded by modifying them so that they fall in levels that are determined in advance relative to the original value. In either case, one bit is embedded in one frequency component, and the secrecy of the embedded information can be maintained by using the primary mask and the convolution mask. However, the embedded information can not survive the performance of data processing, such as compression/decompression and the addition of random noise to each frequency component.

[0012] The delimiter of the message start is a sign consisting of relatively many bits, and is used to detect the delimiting of windows in detected bits and a message start point. According to the specifications for these patents, 64 bits are embedded in one window of 128 samples, and a 1024 bit sign is obtained for 16 windows. Since it is highly improbable these signs will be identical, a specific 1024 bit sign can be employed as the delimiter of the message start. To search for the delimiter of the windows and the message start point, the window start point is shifted one sample at a time until the delimiter of the message start is detected. With this method, if the embedded information is long, the load imposed by the search for the message start point will be increased, and this method can not cope with a request for re-synchronization due to the loss or the insertion of data that frequently occurs during a digital to analog conversion.

[0013] It is, therefore, one object of the present invention to provide a method and a system for embedding additional information, such as copyright information, in audio data, so that a modification in the sonic quality due to the embedding is imperceptible to human beings, and for maintaining the embedded information and precisely detecting it after audio data processing has been performed at a level whereat the sonic quality was not drastically deteriorated.

[0014] It is one more object of the present invention to provide a method and a system whereby additional information can be embedded in audio data and can be detected, while a high sonic quality and high robustness are maintained.

[0015] It is another object of the present invention to provide a method and a system for transforming samples of audio data into frequency components, and for manipulating the obtained data in the frequency domain to embed additional information.

[0016] It is an additional object of the present invention to provide a method and a system, for embedding additional information in audio data, that is robust against destruction during the data processing, such as the compression/decompression of data and the addition of random noise to individual frequency components.

[0017] It is a further object of the present invention to provide a method and a system for embedding additional information in audio data and detecting the embedded information, and for reducing the load imposed by the search for the embedded additional information.

[0018] It is one further object of the present invention to provide a method and a system for embedding additional information in audio data and for detecting the embedded information, and for coping with a request for re-synchronization due to the loss or the insertion of data that occurs frequently in digital to analog conversion.

[0019] To achieve the above objects, according to one aspect of the present invention, provided are an “embedding system,” for embedding additional information, such as copyright information, in uncompressed digital audio data, so that a change in sonic quality is imperceptible to human beings; and a “detection system,” for, even when data compression/decompression or a trimming procedure has been performed for the audio data, determining whether additional information has been embedded and detecting embedded information.

[0020] Uncompressed digital audio data for each channel is constituted by a sequence of integers, called a sample. For the audio data provided by CDs, each channel is constituted by 44,100 16-bit samples per second. According to this invention, information is embedded and is detected at the frequency domain, so that a psychoacoustic model can be employed. Therefore, in the embedding system and the detection system of the present invention, the audio samples are divided into pieces having a constant length, and individual delimited samples pieces are transformed in the frequency domain. Each interval of the pieces of samples to be transformed is called a frame.

[0021] The additional information embedding processing according to the present invention is shown in FIG. 4. At step 410 a mask that defines for each frequency the phase of an embedding signal is employed to embed bit information corresponding to the additional information and a synchronization signal in the frequency component obtained from the individual frames of transformed audio data in a frequency domain, and the audio data in the frequency domain is invertedly transformed into the audio samples in time domain. In the embedding process, the frames do not overlap each other, and successive frames may not be adjacent. When the trimming robustness is required, additional information is repetitively embedded.

[0022] The additional information detection processing is shown in FIG. 5. At step 510 the searching of audio data samples is performed to find a start point of a frame. At step 520, when it is determined that additional information has been embedded, a detection mask is employed to detect a bit embedded in a frequency component. At step 530 a point whereat the cycle of additional information repetitively embedded has been started is searched for, and the embedded additional information is reproduced.

[0023] According to the present invention, provided are a method and a system for embedding, in each frame, information that is aurally imperceptible to human beings but is robust, and for detecting the embedded information; a frame synchronization method and system for searching for the correct frame start and end point before embedded information is detected; and a message synchronization method and system for searching for the bit cycle start and end points in order to reproduce bits (message) by using the bit information detected in each frame.

[0024] Specifically, to embed additional information in audio data, first the audio data are transformed into frequency components. Based on the audio data, the level of modification for each frequency component is determined to be one in which additional information can be embedded in the audio data, and a mask used for embedding additional information is generated. Then, the additional information is embedded, using that mask, within the modification level of the frequency component obtained from the transformed audio data. Finally, the transformed audio data in which the additional information is embedded are invertedly transformed into audio data in time domain. To detect additional information embedded in audio data, synchronization detection means is provided for transforming the audio data into frequency components, for producing a mask used for the detection of additional information and for obtaining synchronization for the detection of additional information. Then, using this mask the additional information in the transformed audio data is synchronously detected from the transformed audio data.

[0025] FIG. 1 is a block diagram illustrating an additional information embedding apparatus according to the present invention.

[0026] FIG. 2 is a block diagram illustrating an additional information detection apparatus according to the present invention.

[0027] FIG. 3 is a diagram illustrating an example hardware arrangement of a system used for embedding/detecting additional information.

[0028] FIG. 4 is a flowchart showing the additional information embedding processing performed according to the present invention.

[0029] FIG. 5 is a flowchart showing the additional information detection processing performed according to the present invention.

[0030] FIG. 6 is a flowchart showing the processing performed to embed a signal in a frame.

[0031] FIG. 7 is a flowchart showing the processing performed to detect a signal embedded in a frame.

[0032] FIG. 8 is a flowchart showing the processing performed to search for a frame start point using the phase.

[0033] FIG. 9 is a flowchart showing the processing performed to estimate a time compression/expansion ratio due to the embedding of a signal for which the phase is aligned.

[0034] FIG. 10 is a diagram showing the difference between the conventional method and the method of the present invention that is employed to embed sets of information for which different intervals are established.

[0035] FIG. 11 is a conceptual diagram of the procedure employed when frames where the message embedding cycle starts are used for the interpretation of bytes, words and messages.

[0036] FIG. 12 is a diagram for explaining the method used to estimate the start point of a message by adding specific bits.

[0037] FIG. 13 is a diagram for explaining the method used to estimate the start point of a message while taking into account the detection reliability of a frame.

[0038] FIG. 14 is a diagram for explaining the method used for estimating the start points of messages by adding specific bits.

[0039] FIG. 15 is a diagram showing an example in which sequences having 5-bit lengths are employed to embed 3-bit information.

[0040] A more detailed explanation will be given for the methods and the systems for embedding additional information in audio data and for detecting the embedded information.

[0041] [Embedding/detection Methods and Systems for Each Frame]

[0042] A frame serves as a unit of samples for a transformation in a frequency domain. The method and the system used for the embedding and the detecting of information for individual frames will now be described. In order to effectively employ a psychoacoustic model, the number of samples of a frame that is considered reasonable is greater than 128, as is described in U.S. Pat. Nos. 5,613,004 and 5,687,236, and actually ranges from about 512 to 2048.

[0043] In the following explanation, an n-th sample in an m-th frame of uncompressed audio data is denoted as Am(n), and a component of frequency k obtained by transforming the m-th frame into frequency components is denoted as Fm(k). When there is no interval between the successive frames and when the number of samples in a frame is N, the following relationship is established.

[0044] A_m(n)=A_Nm+n  Expression 3

[0045] Since the original sample is a real number, even though the individual frequency components are complex numbers that each consist of a real part and an imaginary part, Fm(k) and Fm(N−k)=Fm(−k) are complex conjugates. In other words, the number of independent frequencies is half the number of samples in a frame, i.e., N/2. The complex number is regarded as a two-dimensional vector, and formally the inner product is defined by using the following expression.

[0046] α·β=α_xβ_x+α_yβ_y  Expression 4

[0047] This expression is employed in the following description. Subscripted x and y denote a real part and an imaginary part. At this time, according to the definition,

[0048] α·α=|α|²·

[0049] Assume that the embedding system and the detection system employ in common a set of complex numbers Mp(k) that are pseudo-random numbers relative to each frequency k, and are hereinafter called masks. However, since the embedding mask is a real number in the time domain, Mp(k) and Mp(−k) are complex conjugates. In addition, when Mp(k) consists only of complex numbers having an absolute value of 1, it is assumed that its phase is pseudo-random oriented. If Mp(k) is a pure imaginary number, it is pseudo-random along the imaginary axis. That is, in accordance with the purpose, assume that an adequate limitation is provided for the available range represented by Mp(k).

[0050] [Signal Embedding in a Frame]

[0051] The processing according to the present invention for embedding a signal in a frame is shown in FIG. 6. At step 610 audio samples are divided into frames. At step 620 the individual frames are transformed into frequency components. At step 630 the obtained frequency components are employed to calculate a psychoacoustic model, and a modification level change that is imperceptible to human beings is obtained for each frequency. At step 640, the values of the frequency components are modified as much as possible at the obtained modification level in the directions of the masks acquired by pseudo-random numbers, and bit information, sign information and a synchronization signal are embedded. If the robustness of embedding information is more important than is the sonic quality, the inner products of the frequency components and the individual masks are calculated. When an inner product is smaller than a threshold value required for the robustness, the modification amount in the frequency component is extended until it equals the threshold value, even when the manipulation amount to frequency component exceeds the maximum modification level of the psychoacoustic model. If the sonic quality is more important and the inner product is greater than the threshold value, the modification amount is reduced until it equals the threshold value. When the frequency components have been changed, at step 650 the obtained frequency components are invertedly transformed for retrieve the audio samples from the frequency domain to the time domain. When the frequency component Fm(k) obtained by the transformation of each frame is changed as follows:

[0052] F′_m(k)=F_m(k)+h_m(k)  Expression 5

[0053] the bit information and the synchronization signal can be embedded. Since the frequency component is a complex number, a modification level hm(k) is also a complex number. The psychoacoustic model, such as an acoustic masking model, is calculated for the frequency distribution Fm, and the frequency level for the modification level hm(k) imperceptible to human beings is obtained for each frequency k. The imperceptible level generally has a two complicated dimensional shape. Therefore, even when the statement that the modification hm(k) is the maximum modification that is imperceptible to human beings, the absolute value |hm(k)| is not always the maximum value and is generally located at the border of the modification levels. The acoustic masking model is used to simulate on a computer a condition wherein the level of modification for each frequency, which is imperceptible to human beings, is changed in accordance with the spectral distribution of frequencies, for example, the condition where, when a large volume of pure noise is present, the modification of the preceding and the succeeding frequencies can not be perceived by human beings.

[0054] The embedding of frequency components in the directions of the masks means that the frequency components are so changed that the inner product as represented by $\begin{array}{cc}{X}_p\left(m\right)=\sum _{k}C\left(F_m^{\prime },k\right)\left(M_p\left(k\right)·F_m^{\prime }\left(k\right)\right)& \text{Expression 6}\end{array}$

[0055] has the desired signs or values. In this case, C(Fm,k) represents the positive normalized constant that depends on the frequency distribution Fm and the frequency k, and is used to improve the reliability of a statistical estimate. When the masks are substantially orthogonal, the change in the frequency component due to the embedding is defined by $\begin{array}{cc}{h}_m\left(k\right)=a\left(k\right)\sum _p{b}_p{M}_p\left(k\right)& \text{Expression 7}\end{array}$

[0056] In this expression, the real constant bp may be determined to be proportional to a desired sign or value for the inner product Xp(m) obtained with each mask; and positive constant a(k) may be adjusted so that it is located at the border of the modification levels that are imperceptible. Since normalized constant C(F′m,k) depends on the embedding results, the processes required to obtain the positive constant a(k) are successively performed.

[0057] The embedding of a bit value is performed for each frame. Therefore, if a modification level value differs between adjacent frames for the reason, for example, that the bit values to be embedded differ, a large discontinuity occurs at delimiters A′_m(N−1) and A′_m+1(0) of the frames, and listeners may regard the generated sounds as noise. Therefore, to prevent the deterioration of the sonic quality, the methods used for changing the frequency components and the shapes of masks are limited so that the modification amount is the same at both ends of all the frames. For example, the modification values and masks for the frequency components are limited to purely imaginary numbers. To limit the modification to the purely imaginary numbers is the same as embedding a signal for a sine wave in each frame in the time domain. As another example, the modifications to the frequency components and the masks are limited to real numbers, and a pair of even-numbered frequencies, {2k, 2k′}, and a pair of odd-numbered frequencies, {2k+1, 2k′+1}, are prepared. In order to cancel each other out at both ends of a frame, the values of the masks and the modification values of the paired frequencies have opposite signs, even though the numerals are the same, as in the following expressions.

[0058] M_p(2k)+M_p(2k′)=0 and h_m(2k)+h_m(2k′)=0  Expression 8

[0059] M_p(2k+1)+M_p(2k′+1)=0 and h_m(2k+1)+h_m(2k′+1)=0  Expression 9

[0060] To limit a modification amount to a real number is the same as embedding a signal for a cosine wave in each frame in the time domain.

[0061] In addition, since orthogonal masks are selected, even when information is sequentially embedded in accordance with different masks, the information embedded in one mask can be prevented from destroying that embedded in another, and the occurrence of a detection error whereof the information embedded in one mask is not detected due to the embedding of information in another mask does not take place. As a result, before audio data are uploaded to the server, copyright information of the data is embedded, and after a consumer downloads the audio data across the Internet, trace information such as a user ID is embedded to prevent the server and user from interfering with each other. For example, frequency k at which the mask value Mp(k) is not zero is set in a limited domain, and another mask is so set that a domain that differs from the above is not zero. Then, the orthogonal masks can be maintained without depending on the normalized constant C(Fm,k). At this time, the frequency area in which the masks are not zero is not set to a single frequency interval, but to a plurality of intervals so that the area covers a large number of frequencies, in order that the sonic quality and the robustness can be improved. According to the MPEG compression process, different parameters (scale factors) are employed for individual frequency bands (scale factor bands) for data compression, so that a parameter for controlling a gain is defined for each frequency band. If the frequency band is simply divided and a single interval is allocated for each mask, the robustness of the mask differs greatly in accordance with the data components. Therefore, the frequency band is divided to provide a width that is smaller than a sub-band that is the unit of compression processing, the frequency bands are so collected that the information embedding frequency bands are not concentrated in the low tone range or in the high tone range, and the masks orthogonal to each other are synthesized.

[0062] As is described above, according to the present invention, the embedding method for each frame, unlike the conventional method, is characterized in that:

[0063] Instead of embedding one bit in one frequency component, to embed information the frequency components in a large area are changed in the directions of the masks.

[0064] Instead of determining in advance a modification amount for each frequency, the frequency distribution obtained by frequency transformation is employed to determine, for each frequency, the maximum modification level that is imperceptible to human beings. Then based on the determined level, the modification amount for each frequency component is defined.

[0065] A limitation is imposed on the modification amounts of masks and the frequency component, so that the modification amount at both ends of each frame in the time domain is nearly zero.

[0066] A frequency band is divided into smaller bands and a plurality of these bands are collected so as to prevent frequency biasing, orthogonal masks are prepared, and multiplex information embedding is enabled whereby information is not destroyed by other information.

[0067] [Detection of Signal for a Frame]

[0068] The signal detection processing for a frame according to the present invention is shown in FIG. 7. At step 710 audio samples are divided into a plurality of frames. At step 720 the frames are transformed into frequency components. At step 730 the inner products of the frequency components and the detection mask are calculated. In order to improve the reliability of the detection of embedded information, as needed, the inner product for one frame is added to another frame in which the same information is embedded by multiplying it using an appropriate sign. That is, if the information was embedded with the same sign, it is added with the same sign, and if the information was embedded with the opposite sign, it is added with the inverted sign. Finally, at step 740 the bit information is detected in accordance with the sign of the inner product, or the mask with which the largest inner product was obtained. For each frame, the inner product of the frequency component Fm(k) and the mask Mp(k) is calculated using the following expression. $\begin{array}{cc}{X}_p\left(m\right)=\sum _{k}C\left(F_m,k\right)\left(M_p\left(k\right)·F_m\left(k\right)\right)& \text{Expression 10}\end{array}$

[0069] Here, the normalized constant C(Fm,k) defines a positive number that is introduced to improve the reliability of the statistical estimate and that depends on the frequency distribution and the frequency. Generally, the frequency spectrum |Fm(k)| is not uniformly distributed, and the amplitude tends to be shortened from that for a low frequency component to that for a high frequency component. A sound generator, or data components such as the melody of the music, cause the spectral distribution frequency to fluctuate greatly. To improve the reliability of the statistical estimate, the square of the statistical average <Xp(m)>, when the frequency component is modified to the maximum level that is imperceptible to a human being, is greater than the statistical average square <Xp(m)²> obtained when no information is embedded. Since $\begin{array}{cc}⟨X_p\left(m\right)⟩=\sum _{k}C\left(F_m,k\right)a\left(k\right){\left|M_p\left(k\right)\right|}^2& \text{Expression 11}\\ ⟨{\left(X_p\left(m\right)\right)}^2⟩=\sum _k{\left(C\left(F_m,k\right)\right)}^2{\left(M_p\left(k\right)·F_m\left(k\right)\right)}^2& \text{Expression 12}\end{array}$

[0070] are established, the normalized constant with which the reliability of the statistical estimate is improved is obtained by $\begin{array}{cc}C\left(F_m,k\right)=\frac{\mathrm{ca}\left(k\right){\left|M_p\left(k\right)\right|}^2}{⟨{\left(M_p\left(k\right)·F_m\left(k\right)\right)}^2⟩}& \text{Expression 13}\end{array}$

[0071] At this time, the following expression is established. $\begin{array}{cc}\frac{{⟨X_p\left(m\right)⟩}^2}{⟨{\left(X_p\left(m\right)\right)}^2⟩}\leqq \sum _k\frac{{\left(a\left(k\right)\right)}^2{\left|M_p\left(k\right)\right|}^4}{⟨{\left(M_p\left(k\right)·F_m\left(k\right)\right)}^2⟩}& \text{Expression 14}\end{array}$

[0072] wherein c defines an arbitrary constant, and a(k) is a maximum modification level that is imperceptible to human beings.

[0073] The denominator <(Mp(k)·Fm(k))²> of the normalized constant varies, depending on the statistical average as being a primary set. When a model is employed wherein the frame start point fluctuates, $\begin{array}{cc}⟨{\left(M_p\left(k\right)·F_m\left(k\right)\right)}^2⟩=\frac{1}{2}{\left|M_p\left(k\right)\right|}^2{\left|F_m\left(k\right)\right|}^2& \text{Expression 15}\end{array}$

[0074] can be used to obtain an approximate value, and

[0075] C(F_m,k)=a(k)/|F_m(k)|²  Expression 16

[0076] is acquired. However, since the denominator should not be zero, when |Fm(k)| is smaller than a(k), it is replaced by it, or it is assumed that the following expression is established.

[0077] C(F_m,k)=a(k)/{|F_m(k)|²+a(k)²}  Expression 17

[0078] The upper limit that is set due to the modification or the replacement of the expression is an adequate approximate limit, when it is taken account that, after information has been embedded, compression/decompression causes a random modification such that human beings can not perceive a frequency change. When a simple auditory model wherein a(k) is proportional to |Fm(k)| is further employed,

[0079] C(F_m,k)=1/|F_m(k)|  Expression 18

[0080] is established, and when |Fm(k)| is smaller than the threshold value, it is replaced by it. To examine the reliability of the obtained inner product, first, $\begin{array}{cc}{S}_p\left(m\right)=\sum _k{C\left(F_m,k\right)}^2{\left(M_p\left(k\right)·F_m\left(k\right)\right)}^2& \text{Expression 19}\end{array}$

[0081] is calculated, and it is assumed that

[0082] x_p(m)=X_p(m)/S_p(m)^½  Expression 20

[0083] follows the normalization distribution, so that a false positive error can be calculated.

[0084] Even when all the samples Am(n) are changed to −Am(n) at the same time, the sonic quality of the audio data is unchanged. This data processing is called the inversion of the polarity.

[0085] When apparatuses are connected together to exchange data, the inversion of the polarity occurs in accordance with the apparatus combination. Therefore, if a bit is defined simply in accordance with the inner product Xp(m) or the sign of Xp(m), the audio data is easily deteriorated by the inversion of the polarity. The two following methods can be employed as countermeasures:

[0086] a method for preparing a sign mask and a bit mask, and for retrieving a bit value by using the sign of the result obtained by multiplying the inner product of the bit mask by the inner product of the sign mask; and

[0087] a method for preparing a mask for bit value 0 and a mask for bit value 1, and for retrieving a bit value in accordance with the mask with which the larger inner product is obtained.

[0088] The second method can be regarded as a modification of the first method.

[0089] According to the second method, two masks M0(k) and M1(k) are employed to embed one-bit information. The inner products X₀(m) and X₁(m) are obtained, and the following is established.

[0090] If |X₀(m)|>|X₁(m)|, bit 0.

[0091] If |X₀(m)|<|X₁(m)|, bit 1.  Expression 21

[0092] Specifically, M₀(k) represents the mask for bit 0, and M₁(k) represents the mask for bit 1. However, this method can be regarded as one of the methods for employing the sign mask Ms(k) and the bit mask Mb(k). That is, when

[0093] M_s(k)=M₀(k)+M₁(k)

[0094] M_b(k)=M₁(k)−M₀(k)  Expression 22

[0095] is established,

[0096] X_s(m)=X₀(k)+X₁(k)

[0097] X_b(m)=X₁(k)−X₀(k)  Expression 23

[0098] is also established for the corresponding sign inner product Xs(m) and bit inner product Xb(m). Thus,

[0099] X_s(m)X_b(m)=(X₁(m)−X₀(m))(X₀(m)+X₁(m))  Expression 24

[0100] is obtained. As is apparent from this expression, it can be assumed that if Xs(m) is positive, bit 0 is defined with Xb(m) being negative and bit 1 is defined with Xb(m) being positive, and that if Xs(m) is negative, bit 1 is defined with Xb(m) being negative, and bit 0 is defined with Xb(m) being positive.

[0101] A sign mask is a mask, such as the mask Ms(k), that employs the sign of the inner product for that mask to change the interpretation of the inner product for another bit mask. In order to detect the inversion of the polarity and to interpret the bit correctly, a single sign mask is sufficient for one embedding process. The sign of the inner product for the sign mask must be employed to change the interpretation of the signs of the inner products of other bit masks. In addition, since generally the polarity does not fluctuate within a short interval, to improve the robustness, a signal for the sign mask is embedded with small modification, and a signal for the bit mask is embedded with large modification. The signal for the sign mask is accumulated for each frame to improve the robustness. Assuming that the signal for the sign mask is always embedded in the same direction, the inner product and the distribution of the signals are obtained by the following expression and the sum of continuous frames m. $\begin{array}{cc}{X}_s=\sum _m{X}_s\left(m\right)S_s=\sum _m{S}_s\left(m\right)& \text{Expression 25}\end{array}$

[0102] Since the audio data to be detected may fluctuate in the time domain or as a result of the inversion of the polarity, a large number should not be defined as the frame sum. The sign mask may be employed in common with the frame synchronization mask, or in accordance with a determination as for whether a frame is marked or unmarked. The determination whether a frame is marked or unmarked is performed by

[0103] If X_s²/S_s>t², a frame is marked.

[0104] If X_s²/S_s≦t², a frame is unmarked.  Expression 26

[0105] wherein t is a threshold value for the determination. A corresponding false positive error is calculated while

[0106] x_s=X_s/S_s^½  Expression 27

[0107] follows the normalization distribution. As is described above, the sign mask is more effective.

[0108] As is described above, according to the present invention the detection method for the frames differs from the prior art and is characterized in that

[0109] instead of merely calculating for the simple inner products of frequency components and masks consisting of pseudo-random numbers, that inner products are multiplied by the normalized constants, which are determined from the frequency distribution and which depend on the frequencies, to obtain the final inner product;

[0110] instead of defining a bit value simply in accordance with the sign of the inner product, the sign of the inner product of a sign mask is employed to change the interpretation of a bit value of another mask;

[0111] for all the frames, the sign mask is embedded in the same direction, or alternately, with opposite signs, so that the collection of information spans the frames and the reliability of the detection can be improved; and

[0112] the inner products accumulated for the sign mask are employed to determine whether a frame is marked or unmarked.

[0113] [Frame Synchronization Method and System]

[0114] For the process for detecting a signal for each frame, it is assumed that the audio samples are correctly divided into frames when information is embedded. After information has been embedded the audio data may be expanded or compressed in the time domain, or may be trimmed. Therefore, at the beginning of the detection process and when the determination that embedded information is present is less reliable, the time compression/expansion ratio for the audio data must be estimated, and when information is embedded the frame start point must be searched for.

[0115] [Method for Searching for Frame Start Point Using Phase]

[0116] If for each frame a signal is embedded with a sufficiently large modification, the start point for the audio sample is shifted in order to detect one temporary frame from the samples. This frame is transformed into a frequency component, and the location at which the greatest sum for the absolute values of the inner products is obtained for the individual masks is presumed to be the frame start point in the embedding process. If the signal is embedded with small modification in order to prevent the deterioration of the sonic quality due to the embedding of information, it is assumed that, in the embedding system, the same intervals are set for the frames that are continuously arranged, so that in the detection system, the shifting of the same start point can be employed for a plurality of different frames. Thus, instead of determining the start point using a single frame, a start point at which an adequate value can be provided for a plurality of frames at the same time is searched for by shifting one sample at a time, so that when information is embedded the frame start point can be precisely detected. However, in either case, while shifting the position one audio sample at a time an FFT is performed for the individual frames at a tentative start point, so that too much calculation time is required to search for a start point for the synchronization of frames. Therefore, the following processing is performed to reduce the calculation time. The processing according to the present invention for employing phase to search for the frame start point is shown in FIG. 8. In the embedding system, at step 810 audio samples are divided into tentative frames. Then, at step 820 the frames are overlapped and at step 830 they are transformed into frequency components. At step 840 the phase of the synchronization signal mask data (it may be the same as that of a sign mask) is shifted, and data are embedded in each of the frames in the same direction using the same mask. At step 850 the inner product of the transformed frame and the shifted mask is calculated. At step 860 the location at which the inner product is the maximum is defined as the frame start point when information is embedded. If the absolute value of the inner product at that point is not the maximum, program control returns to step 840. Since the same signal is embedded in each frame, so long as audio samples have a width a multiple of the width at the time the signal is embedded, the detection system can detect the information merely by shifting the phase of the mask (step 840), even though the frame delimiter is shifted away from the frame delimiter when the signal is embedded. The shifting of the frames at a distance equivalent to v samples in the time domain means that a 2πkv/N phase shift is performed in the frequency domain, i.e., that the mask Ms(k) is replaced by Ms(k)e^j2πkv/N. Here, j defines an imaginary number unit.

[0117] Specifically, the frame start point when information is embedded is estimated by performing the following steps. (1) The samples Am(n) are collected in accordance with the tentative frame delimiters, and data for one frame $\begin{array}{cc}A\left(n\right)=\sum _m{A}_m\left(n\right)& \text{Expression 28}\end{array}$

[0118] are synchronized. The resultant data are transformed into frequency components by an FFT, so that the frequency component f(k) is obtained.

[0119] (2) Instead of shifting the positions of the frame delimiters, the phase of the synchronization signal mask Ms(k) is shifted, and the inner product $\begin{array}{cc}{S}_{\mathrm{ync}}\left(v\right)=\sum _{k}C\left(F,k\right)\left(M_s\left(k\right)e^{\frac{\mathrm{j2\pi }\mathrm{kv}}{N}}·F\left(k\right)\right)& \text{Expression 29}\end{array}$

[0120] is calculated for each set of v samples to obtain a “v” at which the maximum |Sync(v)| is obtained. Thus, it is assumed that when information is embedded the start point of the frame delimiters is located v samples ahead of the tentative delimiter start point.

[0121] The obtained Sync(v) can be used as a proxy for the inner product of the sign mask, with which the interpretation of the inner product of a mask for the bit mask is changed. Assuming that when information is embedded the tentative delimiter is at the estimated position, the predicted value for Sync(v) is $\begin{array}{cc}{S}_{\mathrm{ync}}\left(v\right)=\sum _k\lambda \left(k\right)C\left(F,k\right)a\left(k\right){\left|M\left(k\right)\right|}^2\cos \left(\frac{2\pi \mathrm{kv}}{N}\right)& \text{Expression 30}\end{array}$

[0122] wherein k(k) defines a fluctuation due to deterioration following embedding, and more nearly approaches 1 as the occurrence of deterioration is reduced. Since information embedded with a large modification tends to be lost during compression/decompression, a further approximate value for Sync(v) is obtained by calculating the following expression. $\begin{array}{cc}\begin{array}{c}{S}_{\mathrm{ync}}\left(v\right)={\lambda }_{\mathrm{effective}}\sum _{k=k_{\min }}^{k_{\max }}\cos \left(\frac{2\pi \mathrm{kv}}{N}\right)\\ =\frac{\begin{array}{c}{\lambda }_{\mathrm{effetive}}\cos \left(\frac{\pi \left(k_{\min }+k_{\max }\right)v}{N}\right)\\ \sin \left(\frac{\pi \left(k_{\max }+k_{\min }+1\right)v}{N}\right)\end{array}}{\sin \left(\frac{\pi v}{N}\right)}\end{array}& \text{Expression 31}\end{array}$

[0123] Specifically, it is ordinarily expected that Sync(v) will fluctuate around the adequate v, and that its resolution will be N/(kmax−kmin+1). Therefore, it is preferable that the frequencies at which the synchronization signal mask is not zero be distributed in a large domain, and that at the least the frequency for the bit mask should also be employed for the synchronization signal mask.

[0124] [Method for Estimating the Time Compression/expansion Ratio, Due to Embedding of a Signal Having an Aligned Phase]

[0125] In FIG. 9 is shown the processing for estimating the compression/expansion ratio, due to embedding of a signal having an aligned phase. In the process for estimating the frame delimiting position when information was embedded, it is assumed that there is no compression or expansion of audio data in the time domain. The time compression/expansion ratio is estimated in the following manner to detect an embedded signal when the audio data are expanded or reduced in the time domain. The embedding system and the detection system are notified that one or a plurality of specific frequencies exist. At step 920 the embedding system transforms the frequency at a level that is imperceptible to human beings, so that for each frame the frequency has the same phase. Then, at step 930 the detection system changes the width of the frame. At step 950 the detection system determines whether the phase of the frequency, which has been modified by employing the modification of a frame width as the time compression/expansion ratio, is aligned for the individual frames. At step 960 the time compression/expansion ratio is defined by using the modification of the frame width whereat the phase of the frequency is aligned. Suppose that the sample frame width N′ is defined as A_m^(N′)(n)=A_N′m+n, and that a corresponding frequency component is defined as $\begin{array}{cc}{F}_m^{\left(N^{\prime }\right)}\left(k\right)=\sum _{n=0}^{N^{\prime }-1}{A}_m^{\left(N^{\prime }\right)}\left(n\right)e^{\frac{-\mathrm{j2\pi }kv}{N^{\prime }}}& \text{Expression 32}\end{array}$

[0126] If the frame width N is modified to frame width N′ in the time domain, the Fm^(N)(k) in the embedding process is only transformed into the Fm^(N′)(k) in the detection process. Therefore, the embedding system and the detection system are both cognizant of specific frequency ki, and the embedding system embeds information so that the phase Fm^(N)(ki)=|Fm^(N)(ki)| is matched as nearly as possible for all the frames m. The detection system modifies the frame width to obtain the frame width N′ in which $\begin{array}{cc}\left|\sum _m\frac{F_m^{\left(N^{\prime }\right)}\left(k_i\right)}{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}\right|& \text{Expression 33}\end{array}$

[0127] is the maximum at frequency ki. The detection system employs the ratio N′/N to obtain the time compression/expansion ratio. When there are a plurality of frequencies ki that both systems are cognizant of, the frame width N′ is calculated for which $\begin{array}{cc}\sum _i\left|\sum _m\frac{F_m^{\left(N^{\prime }\right)}\left(k_i\right)}{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}\right|& \text{Expression 34}\\ \prod _i\left|\sum _m\frac{F_m^{\left(N^{\prime }\right)}\left(k_i\right)}{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}\right|& \text{Expression 35}\end{array}$

[0128] is the maximum, and the ratio N′/N is used to calculate the time compression/expansion ratio.

[0129] A modification of this processing will now be described. The embedding system and the detection system are notified that one or more specific frequencies exist. The embedding system modifies the frequency within a modification level that is imperceptible to human beings, so that for each frame the frequency has substantially the same phase and amplitude. The detection system modifies the width of the frame, and determines whether the phase of the frequency, which is modified by using the frame width change as the time compression/expansion ratio, is aligned for each frame. The time compression/expansion ratio is defined as the modification of the frame width in which the phase and amplitude are aligned the best.

[0130] Assume that the embedding system and the detection system both are cognizant of specific frequency ki. The embedding system embeds the information so that the frequency component Fm^(N)(ki) is matched as nearly as possible for all the frames m. The detection system changes the frame width to obtain the frame width N′ in which $\begin{array}{cc}\frac{\left|\sum _m{F}_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}{\sqrt{\sum _m{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}^2}}& \text{Expression 36}\end{array}$

[0131] is the maximum at the frequency ki, and employs the ratio N′/N to obtain the time compression/expansion ratio. If the both systems are cognizant of a plurality of frequencies ki, the frame width N′ is calculated in which $\begin{array}{cc}\sum _i\left|\frac{\sum _m{F}_m^{\left(N^{\prime }\right)}\left(k_i\right)}{\sqrt{\sum _m{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}^2}}\right|\mathrm{or}& \text{Expression 37}\\ \prod _i\left|\frac{\sum _m{F}_m^{\left(N^{\prime }\right)}\left(k_i\right)}{\sqrt{\sum _m{\left|F_m^{\left(N^{\prime }\right)}\left(k_i\right)\right|}^2}}\right|& \text{Expression 38}\end{array}$

[0132] is the maximum, and the ratio N′/N is used to acquire the time compression/expansion ratio.

[0133] [Message Synchronization Method and System]

[0134] The message (additional information) must be repetitively embedded in audio data so that it can survive in the trimming of the audio data at any location. Therefore, when bit information is retrieved from the frame, the locations of the head and the end of the bits of the repetitive message must be estimated. That is, the head of the message in the embedding system and the head of the message in the detection system must be correctly synchronized.

[0135] [Method for Employing Frames for Delimiting the Interpretation of Bytes, Words, and a Message]

[0136] FIG. 11 is a conceptual diagram of the arrangement when frames are employed for delimiting the interpretation of bytes, words or a message. In order to insert delimiters for the interpretation of bytes, words and a message in detected bits, the embedding system embeds a synchronization signal in the same direction in all frames except the frame where a message embedding cycle starts, and either embeds a synchronization signal in the opposite direction in the frame where the message embedding cycle starts, or does not embed any signal. When a synchronization signal is embedded in a frame in a direction opposite to that of the others, or when no synchronization signal is embedded therein, the detection system assumes that frame is a frame where the message embedding cycle starts.

[0137] The synchronization signal may be embedded by using the frame synchronization signal mask or the sign mask. In this case, the frame synchronization signal or the sign signal is determined by using the sum of the inner products obtained for a sufficiently large number of frames, so that the affect imposed by the frames where the message embedding cycle starts can be ignored.

[0138] The embedding system embeds the same synchronization signal as often as possible in frames other than the frames where the message embedding cycle starts, and the detection system performs a statistical estimate to ascertain whether the synchronization signal of the detected frame falls in a range in which the synchronization signal of another frame is fluctuating, so that a reliable estimate can be acquired. The detection system is cognizant of the size of the interval separating the frames that are embedded by the embedding system, and searches through the interval of a frame that differs considerably from the others. Therefore, even when the synchronization signal is embedded with relatively small modification to maintain the sonic quality, the synchronization signal can be detected. So long as the embedding system and the detection system both are cognizant of the sizes of the intervals separating the frames, bit information either may be embedded or may not be embedded in a frame where the message embedding cycle starts.

[0139] [Method for Embedding a Plurality of Types of Information that Have Different Repetition Cycles]

[0140] When a plurality of types of information having different importance are to be embedded in audio data, their trimming robustness need not be the same. Important information should be frequently repeated and detected, even when a trimmed interval is short.

[0141] In order to embed and detect a plurality of types of information that have different trimming robustness, the embedding system and the detection system both are cognizant of the length of the bit unit employed for dividing the information, and the number of divisions. In the embedding process, the information is divided into lengths corresponding to that of the bit unit, and the divided bits are retrieved from the information and arranged to prepare information having the equal bit lengths (called synthesized units). The first bit of the first synthesized unit of information is embedded in the frame where the message embedding cycle starts, and the remaining bits are embedded in the succeeding frame.

[0142] For example, assume that the information bits are A, B and C and that they are divided as in

[0143] A₀A₁

[0144] B₀B₁B₂

[0145] C₀C₁C₂C₃  Expression 39

[0146] The embedding system arranges the bits that are divided as in

[0147] A₀B₀C₀A₁B₁C₁A₀B₂C₂A₁B₀C₃A₀B₁C₀A₁B₂C₁A₀B₀C₂A₁B₁C₃A₀B₂C₀A₁B₀C₁A₀B₁C₂A₁B₂C₃  Expression 40

[0148] and uniformly embeds a plurality of information sets to be detected and that have different repetition cycles. The first bit of each ApBqCr is embedded in the frame where the message embedding cycle starts. In FIG. 10 is shown one difference between the method of the present invention and the method of the conventional method when information sets that have different repetition cycles are embedded.

[0149] [Method for Estimating the Start Point of a Message by Adding Specific Bits]

[0150] Assume that the embedding system and the detection system both are cognizant of specific bits (called delimiting bits) that are longer than are the bits of additional information and that do not employ cyclic shifting to match themselves. “The bits do not employ cyclic shifting to match themselves” means that the bits are embedded repeatedly and only when they are trimmed at the original delimiter can the original bits be reproduced. For example, “00100010” can be retrieved either when delimiter “,” is inserted following each repetition, as in

[0151] 00100010,00100010,00100010,00100010, . . . , or when “,” is inserted as in 0010,00100010,00100010,00100010,0010 . . .

[0152] Therefore, the “00100010” can not be employed as delimiting bits. While “00100010” internally contains the repeated “0010,” generally bits that include an internal cycle employ cyclic shifting to match themselves, and can not be used as delimiting bits. On the other hand, “00100011,” which does not include an internal cycle can be used as delimiting bits because only with a specific delimiter the original bits can be retrieved from the repeated bits.

[0153] In addition, assume that the embedding system and the detection system are both cognizant of the method employed for determining whether delimiting bits have been added to the bits by an exclusive-OR (XOR) operation. For example, for the delimiting bits “00100011,” whether the delimiting bits have been XORed can be determined by examining the value of the last bit. The determination method is not unique, and in this example, whether the sum of the bits for all the digits is an odd number or an even number can be employed for the determination. FIG. 12 is a diagram for explaining the method used for estimating the start point of a message by adding specific bits.

[0154] The embedding system adds one or more bits to the bits of additional information, prepares expanded bits that are as many as the delimiting bits, and embeds in audio data the expanded bits, as well as the delimiting bits which have been added to every other frame by the performance of an XOR operation. The additional bit is employed to determine whether the delimiting bits have been added.

[0155] By employing an XOR operation, the detection system adds to detected bits the same bits that are shifted a distance equivalent to the length of the delimiting bits, and searches the obtained bits for the delimiters which enable the obtained bits to match the delimited bits. If such delimiters are found, they are defined as the locations of the delimiters in the detected bits. If no such delimiters are found, it is assumed that an error occurred when the bits were detected. Since delimiting bits have been added to every other frame in the delimited bits, bits to which nothing has been added are determined by employing an additional bit, and are defined as retrieved bits.

[0156] Assume that “00100011” are delimiting bits, and that the additional information in the embedding system is seven bits “1101101.” One “0” bit is added to the additional information, and the obtained expanded bits “11011010” are repeatedly embedded. At this time, through an XOR operation the delimiting bits are added to every other frame and are embedded every other embedding repetition:

[0157] 11011010,11111001,11011010,11111001,11011010,11111001, . . .

[0158] In the above example, the delimiter “,” is inserted to make the separations easy to see; actually, however, only a bit value of 0 or 1 is embedded in each frame.

[0159] Then, suppose that the first three bits of the above sequence are lost due to a process that follows the embedding, and the detection system detects the following bits:

[0160] 11010,11111001,11011010,11111001,11011010,11111001, . . .

[0161] These bits are shifted a distance equivalent to the number of the delimiting bits, and the obtained bits

[0162] 11001,11011010,11111001,11011010,11111001,11011010, . . .

[0163] are added by employing the XOR operation. Then, the resultant bits are

[0164] 00011,00100011,00100011,00100011,00100011,00100011, . . .

[0165] Since the obtained bits match the delimiting bits “00100011”, only this delimiter is correct. Then, expanded bits having the additional “0” bit are selected at each delimiter, and the bits “1101101” are obtained as additional information.

[0166] [Method for Estimating the Start Point of a Message While Taking into Account the Reliability of the Detection Performed in a Frame]

[0167] FIG. 13 is a diagram for explaining a method for estimating the start point of a message while taking into account the reliability of the detection performed in a frame. In order to estimate the start point of a message while taking into account the reliability of the detection performed in a frame, the detection system calculates the inner products of the bit mask for two corresponding frames that are determined in accordance with the length of the delimiting bits. The inner products are multiplied by the sign defined in the delimiter bits. Then, a delimiter is searched for at which the sum of the inner products in the succeeding frame is the maximum. Assume that the delimiting bits are defined as {B(m)} and the length is defined as M, and that the inner product in each frame obtained by using the bit mask is defined as Xb(m). Then, $\begin{array}{cc}P\left(v\right)=\sum _{m=0}^{M-1}{\left(-1\right)}^{B\left(m-v\right)}{X}_b\left(m\right)X_b\left(m+M\right)& \text{Expression 41}\end{array}$

[0168] is calculated to obtain the “v” at which the sum is the maximum, and its location is defined as the start point of the message. When the value of the subscript of the delimiting bit exceeds the original range, assume that the range is expanded as B(m+M)=B(m), and that (−1)⁰=1 and (−1)¹=−1. According to this method, even when a bit error occurs in the individual frames, the start point of the message can be obtained.

[0169] Once the start point of the message has been obtained, the bit information is calculated by using the sign of $\begin{array}{cc}X\left(m\right)=\sum _r{\left(-1\right)}^{p\left(r\right)B\left(m\right)}{X}_b\left(\gamma M+m+v\right)& \text{Expression 42}\end{array}$

[0170] that is accumulated concerning the repetitions. Here p(r) is 1 for the repetitions with the delimiting bit being added, or is zero for the repetitions with out the delimiting bit being added.

[0171] [Method for Estimating the Start Points of a Plurality of Messages by Adding Specific Bits]

[0172] In order to embed information sets having different cycles, specific bits that do not match themselves by cyclic shifting (called a delimiting bit segment) are added by an exclusive-OR operation and are embedded, so that the start point of the message can be detected.

[0173] The meaning of “The bits do not match themselves by shifting” is as follows. When arbitrary shift v (0<v<M) is performed for bits {B(m)} having a length M, m, where B(m)≠B(m+v) is established, is always present in a range of 0≦m≦m+v≦M−1. For example, “01011011101111” and “010011000111” are example bits that do not match themselves by shifting.

[0174] Assume that the embedding system and the detection system are notified that there are bits of additional information to be repetitively embedded at short intervals, and that there is a delimiting bit segment that is longer than the bits of the additional information and that is to be repetitively embedded at longer intervals. The embedding system adds a bit to the bits that are repetitively embedded at short intervals, and prepares expanded bits that are as long as the delimiting bit segment. Furthermore, the embedding system adds the other bits to the obtained bits to prepare a synthesizing bit unit. To embed the synthesized bit units in order, the embedding system embeds the delimiting bit segment in the expanded bits by performing an exclusive-OR operation. The additional bit is employed to determine whether the delimiting bit segment is added.

[0175] By employing the XOR operation, the detection system adds to the synthesized bit unit, the same detected bits after they have been shifted the equivalent of the length of the bits. Then, the detection system searches for the delimiter whereat the bits correspond to the delimiting bits. If the delimiter is found, it is defined as the location of the delimiting of the detected bits. If the delimiter is not found, it is assumed an error occurred when the bits were detected. Since at every other delimiter the delimiting bit segment is added to the delimited bits, the additional bit is employed to determine bits to which nothing was added, and that bits are defined as retrieved bits.

[0176] Assume that “00110111” is a delimiting bit segment, and the additional information having a short cycle is seven bits, “1101101.” The embedding system adds one bit “0” to prepare the expanded bits “11011010.” And then the bits “*******” of the other additional information having a long cycle are attached to the expanded bits, so that the bits that constitute the synthesized unit are obtained as “11011010*******.” The asterisk, “*,” can be either 0 or 1. When the synthesized bit unit is to be embedded, the delimiting bit segment is XORed and embedded at the location of every other expanded bit. Thus,

[0177] 11011010*******,11101101*******,11011010*******,11101101***** **, . . .

[0178] are embedded for each frame. While the delimiter “,” is inserted in this case, actually only the bit value 0 or 1 is embedded in the frame.

[0179] Assume that the first three bits are lost from the above sequence due to the processing performed following the embedding, and that the detection system can detect the following bits.

[0180] 11010*******,11101101*******,11011010*******,11101101*******,

[0181] When these bits are shifted the equivalent of the length of the synthesized bit unit, and the obtained bits

[0182] 01101*******,11011010*******,11101101*******, . . .

[0183] are XORed and added. Then, the resultant bits are

[0184] 10111*******,00110111*******,001101111*******, . . .

[0185] Regardless of the values of “******,” this is shorter than the length of the delimiting partial bit segment, so that this sequence can neither match the bits “00110111” nor match itself by shifting. Thus, it is apparent that only the delimiter used for embedding matches “00110111.” Then, when the expanded bits having the additional “0” bit are selected at each delimiter, it is found that the bits of the additional information having the short cycle are “1101101.” Similarly, a short bit sequence “*******” that is attached to the additional information is detected. And the delimiting bit segment having a corresponding length is employed, so that the start point of the additional information can be detected. In FIG. 14 is shown the method used for estimating the start points of a plurality of messages by adding the above specific bits.

[0186] [Embedding a Sign that is not Affected by the Cyclic Shifting]

[0187] The embedding system and the detection system are notified of the number of frames of one cycle where the additional information is assumed to be embedded repeatedly. A set of bits that are rendered equal to each other by cyclic shifting are regarded as having the same values, and bits that have a different value correspond to different additional information. In this manner, the system can embed information and detect the embedded information without being cognizant of where the cycle of bits starts.

[0188] Bits that are not rendered equal to each other by cyclic shifting are, for example,

[0189] {00000},{00001}{00011},{00101},{00111},{01011},{01111}, {11111}.

[0190] The sequences of five bits each can be employed to represent eight states, i.e., 3-bit information. In FIG. 15 is shown an example for employing the above sequences having 5-bit lengths to embed 3-bit information. Assume that S(M) is defined as the number of bits of bit length M that are not rendered equal to each other by cyclic shift. Then, $\begin{array}{cc}S\left(p\right)=2+\frac{\left(2^{p-2}\right)}{p}& \text{Expression 43}\\ S\left(p^{m+1}\right)=S\left(p^m\right)+\frac{\left(2^{p^{m+1}}-2^{p^m}\right)}{p^m}& \text{Expression 44}\end{array}$

[0191] are established for prime number p. For p and q that are relative prime numbers, $\begin{array}{cc}S\left(\mathrm{pq}\right)=S\left(p\right)+S\left(q\right)+\frac{\left(2^{\mathrm{pq}}-2^p-2^q\right)}{\left(\mathrm{pq}\right)}& \text{Expression 45}\end{array}$

[0192] is established.

[0193] The preferred embodiment of the present invention will now be described while referring to FIGS. 1 to 3. FIG. 1 is a block diagram illustrating an additional information embedding apparatus according to the present invention. An input control block 110 transmits, to a frequency transformation block 120, received audio signals as sample values that are arranged in a time series. When the input signal is analog, it is converted into a digital signal. When the input signal is digital, its format is translated and audio samples are transmitted to the block 120.

[0194] The frequency transformation block 120 calculates the frequency components of an audio signal. The block 120 receives a digital audio signal from the block 110, and calculates the frequency components in the signal interval.

[0195] A psychoacoustic model calculation block 130 employs the hearing response of human beings to calculate a level in which a modification is difficult to identify. The block 130 receives the transformed audio signal from the block 120, and calculates, for each frequency component, the affect it has on hearing.

[0196] A pseudo-random number generation block 160 generates pseudo-random numbers consisting of 0 and 1. The generated pseudo-random numbers are transmitted to an additional information embedding mask generation block 170.

[0197] The additional information embedding mask generation block 170 employs the received pseudo-random numbers to generate a mask used to embed additional information. The block 170 correlates the pseudo-random numbers with the phase direction, and generates a two-dimensional vector (mask) on a complex number plane that defines, for each frequency, the phase of an embedded signal.

[0198] Additional information 180 is transmitted as bit information to an additional information embedding block 140. As needed, a bit for the detection of message synchronization is inserted into the bits, and the bits are inverted.

[0199] The additional information embedding block 140 embeds imperceptible additional information in an audio signal. Based on the psychoacoustic model obtained by the block 130, in the direction of the masks obtained by the block 170, the block 140 adds, within an imperceptible level, a signal and a synchronization signal used for time compression/expansion detection to the audio signal transformed by the block 120. As a result, the statistical value of the frequency components of the audio signal is a sign that is defined in accordance with the bit information carried in the additional information 180.

[0200] A frequency inverted transformation block 190 invertedly transforms the above audio signal. The block 190 invertedly transforms the audio signal in which the additional information is embedded by the block 140, and calculates audio samples that are arranged in a time series.

[0201] An output control block 150 outputs the audio samples as audio data. The block 150 performs DA conversion of the audio samples received by the block 190, or adds header information as a file to the audio samples, and outputs the resultant audio signal.

[0202] FIG. 2 is a block diagram illustrating an additional information embedding apparatus according to the present invention. An input control block 210 transmits, to a frequency transformation block 220, received audio signals as sample values arranged in a time series. When the input signal is analog, it is converted into a digital signal. When the input signal is digital, its format is translated and audio samples are transmitted to the block 220.

[0203] The frequency transformation block 220 calculates the frequency components of an audio signal. The block 220 receives a digital audio signal from the block 210, and calculates the frequency component for the signal interval.

[0204] While assuming that additional information is embedded in an audio signal, an additional information synchronization detection block 250 calculates the time compression/expansion ratio of a received audio signal, and detects the start point of an additional information embedding process unit (a frame). The block 250 measures the time compression/expansion ratio of the synchronization signal to detect the time compression/expansion ratio of the audio signal transformed by the block 220. Then, while rotating an additional information detection mask received from an additional information detection mask generation block 280, the block 250 calculates the inner product of the mask and the audio signal whose time compression/expansion ratio is compensated for. The block 250 detects a shifting of the frame start point by employing the rotation angle of the mask at which the inner product is the maximum.

[0205] A pseudo-random number generation block 270 generates pseudo-random numbers consisting of 0 and 1. The generated pseudo-random numbers are transmitted to an additional information detection mask generation block 280.

[0206] The additional information detection mask generation block 280 employs the received pseudo-random numbers to generate a mask that is used to detect additional information. The block 280 correlates the pseudo-random numbers with the phase direction, and generates a two-dimensional vector (a mask) on a complex number plane that defines, for each frequency, the phase of an embedded signal.

[0207] An additional information detection block 260 detects additional information embedded in the audio signal. The block 260 calculates as a statistical value the inner product of the transformed audio signal, for which the time compression/expansion ratio is obtained by the block 250 and the frames are synchronized, and the additional information detection mask obtained by the block 280. If the synchronization of frames must be performed again because of the obtained statistical value, a re-synchronization signal is transmitted to the block 250 to detect the synchronization of additional information.

[0208] An additional information presence detection block 230 employs the obtained statistical value to determine whether additional information is embedded in the audio signal. In order to determine whether the information is intentionally embedded in the audio signal, the block 230 determines whether the statistical value obtained by the block 260 is satisfactorily large.

[0209] An additional information reproduction block 240 reproduces the detected bit information as additional information. When the block 230 determines that the additional information has been embedded, the block 240 correlates the sign affixed to the statistical value with the bits and translates it as bit information. The block 240 then examines the detected bit information to detect a start bit of the additional information, and shifts the detected bit information in accordance with the start bit. When bits are inserted/inverted for the synchronization of a message, the block 240 removes/inverts these bits, and retrieves the additional information.

[0210] In FIG. 3 is shown the general hardware arrangement for a system according to the present invention that is employed for embedding/detecting additional information. A system 100 comprises a central processing apparatus (CPU) 1 and a memory 4. The CPU 1 and the memory 4 are connected by a bus 2 and an IDE controller 25 to a hard disk drive 13 (or to a storage medium drive, such as an MO, a CD-ROM 26 or a DVD 32), which is an auxiliary storage device. Similarly, the CPU 1 and the memory 4 are connected by the bus 2 and a SCSI controller 27 to a hard disk drive 30 (or to a storage medium drive, such as an MO 28, a CD-ROM 29 or a DVD 31), which is an auxiliary storage device, while a floppy disk drive 20 is connected to the bus 2 via a floppy disk controller 19.

[0211] A floppy disk is inserted into the floppy disk drive 20, and code or data for a computer program, which interacts with an operating system and issues instructions to the CPU 1 for embedding or detecting additional information to implement the present invention, can be stored on the floppy disk or on the hard disk drive 13 (or on the MO, the CD-ROM 26 or the DVD 32), and in a ROM 14. This computer program is executed by loading it into the memory 4. The code for the computer program can be compressed, or can be divided into a plurality of segments for storage on a plurality of storage mediums.

[0212] The system 100 further comprises user interface hardware components such as a pointing device (a mouse or a joystick) 7 and a keyboard 6 for entering data, and a display device 12 for providing visual data for a user. Furthermore, a printer can be connected to the system 100 via a parallel port 16, or a modem can be connected via a serial port 15. For communicating with another computer, the system 100 can be connected to a network via the serial port 15 and a modem, or via a communication adaptor 18 (an ethernet or a token ring card). Additional information to be embedded can be fetched from an external network via the communication adaptor 18, or reproduced additional information can be transmitted externally via the network. A remote transceiver is connected to the serial port 15, or to the parallel port 16, for exchanging data with the system 100 using infrared rays or wires.

[0213] A loudspeaker 23 is used to receive, via an amplifier 22, an audio signal produced by D/A conversion (digital/analog conversion) performed by an audio controller 21, and to output the signal as sound. The audio controller 21 performs the A/D (analog/digital) conversion of audio information received via a microphone 24 in order to introduce external audio data as additional information to be embedded into the system 100.

[0214] As is described above, it would be easily understood that the system 100 of the present invention can be provided by an average personal computer (PC); a workstation, a notebook PC or a palmtop PC; a network computer, a television set or another electric home appliance that incorporates a computer; a game machine having a communication function; a communication terminal having a communication function, such as a telephone, a facsimile machine, a portable telephone, a PHS or a PDA; or a combination of the components described above. It should be noted, however, that these components are merely examples and not all of them are required for the present invention. The additional information embedding/detection blocks in FIGS. 1 and 2 may be constituted by dedicated hardware components.

[0215] According to the present invention, provided are a method and a system for so embedding additional information, such as copyright information, in audio data that a modification in the sonic quality due to the embedding is imperceptible to human beings, and for maintaining the embedded information and precisely detecting it after processing of the audio data has been performed that does not drastically deteriorate the sonic quality. Therefore, copyrighted music recordings that are distributed on CDs or via the Internet can be protected, high quality recorded music can be safely supplied as digital data, and the musical culture can be stably supported.

[0216] It should be understood that the scope of the present invention is not limited to the described embodiment but includes those changes within the spirit and scope of the appended claims.