In-service measurement of transmitter nonlinearities

[0002] The present invention relates generally to the measurement of transmitter parameters, and more particularly to a method and apparatus for in-service measurement of a transmitter's magnitude and phase nonlinearities.

[0003] Nonlinear magnitude and phase responses are common distortions caused by a transmitter power amplifier that affects the quality of a transmitted signal. The gain and phase responses of the amplifier are a function of the input signal magnitude to the amplifier, which can drive the amplifier into nonlinear operation. The distortion present in a received signal from the transmitter is often a mixture of linear distortion, nonlinear distortion, phase noise and additive noise. Correctly separating the nonlinear errors from the other distortions is desired for accurate measurement. However, with in-service measurement, the received signal is data bearing and its content is unknown to the measurement instrument.

[0004] In an 8 level Vestigial Side Band (8-VSB) digital modulation system, for example, the overall system response of the combined transmitter and receiver corresponds to a raised cosine filter to avoid system generated intersymbol interference. The system response is implemented with nominally identical root raised cosine filters in the transmitter and in the receiver. The information bearing digital data stream is randomized for spectrum spreading over the bandwidth of the frequency channel. The randomized data is forward-error-corrected (FEC) coded and interleaved. The data is trellis encoded as an 8-level (3-bit) one dimensional constellation with the outputs of the trellis coder referred to as symbols that are one of eight symmetric odd-valued integer levels from −7 to +7 units. To aid synchronization in low signal to noise and/or high multipath situations, segment and field syncs are inserted in the 10.76 Msymbols/sec symbol stream as well as a small pilot tone at the carrier frequency generated by offsetting the real or I channel of the complex signal containing the data and the sync pulses by 1.25 units. At the transmitter, the composite signal passes through a root raised cosine filter and modulates an intermediate frequency carrier signal which is up-converted to an RF frequency for transmission at the desired channel frequency. The offset causes the pilot tone to be in-phase with the I channel carrier frequency. Alternately, the composite signal may directly modulate the RF carrier.

[0005] Synchronous demodulation may be used to detect the eight constellation decision levels. However, the constellation of the transmitted symbols may bend or stretch to form certain curvatures due to the nonlinear distortions in the transmitter. Therefore, for estimation of constellation decision levels, a conventional slicing method is inadequate for symbol decision in the presence of large nonlinear distortions. For measurement purposes, attempting to determine an ideally transmitted signal from the received signal with large transmitter phase and/or magnitude nonlinearity would not be reliable using the conventional slicing method.

[0006] Also, conventional methods of measuring transmitter nonlinearity that apply the band-limiting transmission system's receiver filter to the received intermediate frequency (IF) signal would alter the characteristics of the nonlinear distortions. Nonlinear and linear transfer functions are usually not commutative, as stated in FIG. 1. That means the nonlinear function observed from the demodulated baseband signal (with the use of the transmission system's receiver filter) is different from the original nonlinear function of the power amplifier. It is difficult to derive the original nonlinear function from what is observed from the demodulated baseband signal, especially with a randomized digital signal. Also, strong nonlinearity causes signal spectrum spreading. The transmission system's receiver filter will significantly attenuate the out-of-band portion of the spread spectrum signal with the loss of spectral information characterized by the nonlinear distortions.

[0007] To determine the causes of poor signal quality of the transmitted signal and to provide pre-correction characteristics for the transmitter, accurate measurement of transmitter nonlinearities is very useful. However, the transmitter nonlinearities in the received signal are often combined with other distortions, such as linear distortions, carrier phase jitter, and additive noise. Due to the difficulty of separating various errors, nonlinearity is traditionally measured in an out-of-service mode. An out-of-service mode requires turning off the broadcast channel(s) (i.e. removing the modulations), which causes the loss of nonlinearity characteristics caused by the presence of the digital signal as well as other distortions.

[0008] What is needed is a method and apparatus for in-service measurement of transmitter magnitude and phase nonlinearities of a received signal, where the signal has a mixture of linear distortions, nonlinear distortions, phase noise and additive noise present, that uses reliable estimation of constellation decision levels and preserves original spectral information.

[0009] In accordance with the illustrated preferred embodiment of the invention, a system is disclosed which provides for in-service measurement of magnitude and phase nonlinearities of a modulated received signal that has a mixture of linear distortions, nonlinear distortions, phase noise and additive noise present, while using reliable estimation of constellation decision levels which preserves original spectral information.

[0010] In the preferred embodiment of the invention, the modulated received signal is an 8-VSB digital television signal having symbol data at 10.76 Msymbols/sec modulating a carrier signal. Prior to signal measurement, transmitter nonlinearities in a received signal are separated from linear distortions. This is performed by first demodulating the modulated received signal to baseband signal samples, filtering the signal samples with a transmission system receiver filter, time-aligning the signal samples to the symbol instances, scaling and linearly equalizing the signal samples to produce filtered signal samples. The timing parameters, scaling factor, and equalization coefficients of that process are applied to the demodulated signal samples without applying the transmission system receiver filter to produce unfiltered signal samples. The signal samples representing the transmitted symbols, which may be other than 8-VSB, are then estimated for purposes of generating reference signal samples representative of transmitted symbols. This is performed by dynamically estimating constellation decision levels from either the filtered or unfiltered signal samples. Magnitude and phase nonlinearities are then measured by comparing the unfiltered signal samples with the locally generated reference signal samples. While doing so, nonlinear distortions are distinguished from other noise-like distortions by using the systematic nature of transmitter nonlinearities and the random nature of other distortions by computing a weighted, least-square based polynomial regression on the error data between the unfiltered signal samples and the reference signal samples.

[0011] The objects, advantages, and other novel features of the present invention are apparent from the following detailed description when read in conjunction with the appended claims and attached drawing.

[0012] FIG. 1 is a diagram demonstrating the non-commutative operation between linear and non-linear functions.

[0013] FIG. 2 is a block diagram illustrative of a transmission receiver system incorporating apparatus and method for in-service measurements of transmitter nonlinearities according to the present invention.

[0014] FIG. 3 is a representative constellation diagram of an 8-VSB signal showing actual and corresponding ideal vectors plus their trajectories over time for the purpose of understanding the apparatus and method for in-service measurements of transmitter nonlinearities according to the present invention.

[0015] FIG. 4 is an improved demodulator used with the system for in-service measurement of transmitter nonlinearities according to the present invention.

[0016] FIG. 5 is a block diagram of a system for in-service measurement of transmitter nonlinearities according to the present invention.

[0017] FIG. 6 is a block diagram of the improved reference signal generator in the system for in-service measurement of transmitter nonlinearities according to the present invention.

[0018] FIG. 7 is a graphical representation illustrating the implementation of the dynamic slicer the system for in-service measurement of transmitter nonlinearities according to the present invention.

[0019] FIG. 8 is a more detailed block diagram of the system for measuring transmitter nonlinearities illustrating the measurement of transmitter magnitude nonlinearity according to the present invention.

[0020] FIG. 9 is a more detailed block diagram of the system for measuring transmitter nonlinearities illustrating the measurement of transmitter phase nonlinearity according to the present invention.

[0021] FIG. 10 is a representation of a graphic display of measurement results for magnitude nonlinear error.

[0022] FIG. 11 is a representation of a graphic display of measurement results for phase nonlinear error.

[0023] In the following detailed description numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances well known methods, procedures, components, and circuits have not been described in detail so as not to obscure the present invention.

[0024] Part of the description will be presented in terms of operations performed by a computer system, using terms such as data, values, signal samples, flags, numbers and the like, consistent with the manner commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art. As is well understood by those skilled in the art, these quantities take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined and otherwise manipulated through mechanical and electrical components of the computer system; and the term computer system includes general purpose as well as special purpose data processing machines, systems, and the like, that are stand alone, adjunct or embedded.

[0025] Additionally, various operations will be described as multiple discrete steps in turn in a manner that is most helpful in understanding the present invention, however, the order of description should not be construed as to imply that these operations are necessarily order dependent, in particular, the order of their presentation.

[0026] The present invention will be described in relation to an 8-VSB digitally modulated RF signal. It is understood that other digitally modulated RF signals may be used with the present invention without departing from the scope of the appended claims. For example, the proposed European standard for terrestrial digital television broadcasting, DVB-T using an OFDM format, calls for generating multiple carriers over a frequency band with each carrier containing a piece of data. The receiver performs a Fourier Transform of the received signal to retrieve the data. Such a system is not required to have transmitter and receiver filters. An 8-VSB RF signal has data symbols that modulate the RF carrier at a symbol frequency at approximately 10.76 Msymbols/sec. Further, the invention is described using baseband filtered, unfiltered and reference signal samples that have been processed through linear equalization processes. The in-service measurement of transmitter nonlinearities of the present invention may also be implemented where a known training sequence is used for generating the reference signal samples or are provided as the reference signal samples. Also, the invention may be practiced using IF signal samples.

[0027] FIG. 2 shows a representative block diagram of a transmission system receiver 10, as could be used in a measurement instrument, digital television receiver or the like. The receiver 10 includes a hardware front end 12 receiving a digitally modulated RF input signal 14. The RF signal 14 is down converted to a digitally modulated intermediate frequency (IF) signal by RF hardware down converter circuitry 16 that generally includes one or more mixers in the IF signal path. Each mixer is driven by a local oscillator. An analog-to-digital (A/D) converter 18 receives the digitally modulated IF signal and converts the signal to digital data values that are stored in memory 20. In the preferred embodiment, the IF signal is sampled at four times the 8-VSB symbol frequency. Memory 20 includes both RAM, ROM and cache memory with the RAM memory storing volatile data, such as the data values representative of the IF signal and the like. A data and control bus 22 couples memory 20 to a controller 24, such as PENTIUM® microprocessor, manufactured and sold by Intel, Corp., Santa Clara, Calif. The data and control bus 22 may also be coupled to the front end hardware 12, a display device 26, such a liquid crystal display, cathode ray tube or the like, and a front panel 28 with buttons, rotatable knobs and the like and/or control entry devices, such as a keyboard and/or mouse. A mass storage unit or units 30, such as a hard disk drive, a CD ROM drive, a tape drive, a floppy drive or the like, that read from and/or write to appropriate mass storage media, may also be connected to the data and control bus 22. A software demodulator 34 is representatively shown connected to the bus 22 and coupled to memory 20. The software demodulator 34 executes various processes that are performed by the controller 24 using processing routines stored in memory 20 including program instructions performing the in-service measurement of transmitter nonlinearities. Data flow lines 36 connect the memory 20 with the software demodulator 34 for showing the movement of data from the memory 20 to the software demodulation process and back. The signal sample output from the various demodulator processes are stored in memory 20. The program instructions may be stored and accessed from the ROM memory 20 or from the mass storage media of the mass storage unit 30. The receiver 10 in the preferred embodiment of the invention is a PC based system controlled under WINDOWS® 95 operating system, manufactured and sold by Microsoft, Corp., Redmond, Wash. Graphical plotting software, such as PROESSENTIALS™ Graphic Display Software, manufactured and sold by Gigasoft, Inc. Keller Tex., is used for generating graphical representations of magnitude nonlinearity measurement of the present invention. The nonlinearity measurements may also be exported using an I/O device 35. The apparatus for performing the in-service nonlinearity measurements of the present invention may also be implemented using hardware circuitry performing the same functions as described for the software implementation. Alternately, a hybrid system using multiple controllers, such as controller 32 connected to controller 24 via a serial bus 38, may be used to implement the receiver 10 and the nonlinearity measurement functions of the present invention. The controller 32 is coupled to a separate data and control bus 40. Controller 32 may separately control display device 26 or perform a portion or portions of the software demodulator 34 functions. An I/O device 41 may be coupled to the control bus 40 for exporting the results of the software demodulator function or the nonlinearity measurements. Further the nonlinearity measurement functions may be implemented using both hardware circuitry, as represented by ASIC 42 and software routines performed by the controller 24 or controller 24 and 32.

[0028] Referring to FIG. 3, there is shown a polar plot of 8-VSB signal samples associated with lines representing a constellation diagram of 8-VSB signal samples. The figure will be used to describe some fundamental concepts of the 8-VSB system as well as some of the relationships affecting the nonlinearity measurements. The overall 8-VSB system response of the combined transmitter and receiver utilizes a raised cosine filter to create samples free of system generated inter-symbol interference at symbol times. The system response is implemented with nominally identical root raised cosine filters in the transmitter and in the receiver. The overall impulse response of the 8-VSB system is so constructed that ideally there would be no inter-symbol interference in the in-phase components which would result in 8 distinct vertical lines in the constellation diagram, and that the imaginary part of the response is combined with real part of the spectrum to suppress the unwanted sideband spectrum. Ideally, the real components are landing along the 8 vertical lines of the constellation diagram. The complex 8-VSB symbol data may be visualized on the complex plane as a vector, rotating about the origin, moving from one symbol value to another. The in-phase (real) parts of the symbol data are grouped along the real (I) axis at the appropriate symbol value. The quadrature parts of the symbol data is required to suppress the unwanted sideband and to minimize intersymbol interference and provide smooth transitions between the symbols. This is represented in FIG. 3 by line 44 for received signal samples 82, 84, and 86 of transmitted data having nonlinear magnitude, phase errors and intersymbol interference for future reference.

[0029] The IF signal samples, stored in memory 20, are a record of 8-VSB IF data acquired at four times the 8-VSB symbol rate and denoted as r(n). The IF signal samples are demodulated to baseband signal samples by the improved demodulator 34, although the in-service nonlinearity measurement method disclosed herein could be applied to an IF signal as well. The baseband signal samples are time-aligned so every fourth sample falls at a symbol instance, scaled and linearly equalized by the improved demodulator 34, although equalization may not be necessary if no significant linear distortions are present. Referring to FIG. 4, the incoming IF signal is input to the improved demodulator 34 that first processes the signal samples in a first processing channel 120 to estimate the carrier phase, symbol timing frequency and phase offsets, gain factor, pilot amplitude, and equalizer coefficients. For the same IF record of data the demodulator 34 uses the carrier phase value to down convert 122 the IF signal to base band signal samples, which are stored in memory 20. The baseband signal samples in memory 20 are passed through a digital filter 124 having combined filter coefficients 126 producing a filter response of the transmission system receiver filter, which in the preferred embodiment is a root raised cosine filter, and compensation for linear distortions introduced by the hardware of the front end circuitry 12. A symbol timing synchronizer 128 estimates timing phase and rate offset parameters using the down converted signal samples and the combined filter coefficients 126 and applies the parameters to a resampling filter to time-align the filtered signal samples so that every fourth samples falls at a symbol instance. A signal scaling and pilot level remover 130 receives the filtered, time-aligned signal samples and determines a scaling factor and pilot level that are used to scale the filtered, time-aligned signal samples and remove the pilot level from the signal samples. An equalizer 132 receives the filtered, time-aligned, scaled signal samples and calculates equalizer coefficients that are applied to a equalization filter within the equalizer to produce filtered, time-aligned, scaled, equalized signal samples that are free of linear errors. The filtered, time-aligned, scaled, equalized signal samples from the first processing channel 120 are output from the demodulator 34 as filtered signal samples at four samples per 8-VSB symbol time (Tsymb/4). Alternatively, the filtered, time-aligned, scaled, equalized signal samples may be passed through a down sampler 133 to produce filtered signal samples at two samples per symbol time (Tsymb/2) or samples at symbol time (Tsymb).

[0030] The second processing channel 134 of the improved demodulator 34 receives the down converted baseband signal samples from memory 20. A compensation filter 136 receives the down converted signal samples and the coefficients that compensate for the linear distortion introduced by the front end hardware circuitry 12. The output of the compensation filter are compensated signal samples. A resampling filter 138 receives the compensated signal samples and the timing phase and rate offset parameters from the symbol timing synchronizer 128 and produces compensated signal samples time-aligned to the symbol frequency. A scaler 140 receives the compensated, time-aligned signal samples and the scaling factor from the pilot level estimator 130 and produces compensated, time-aligned, scaled signal samples. An equalization filter 142 receives the compensated, time-aligned, scaled signal samples and the equalizer coefficients from the equalizer 132 and produces compensated, time-aligned, scaled, equalized signal samples that are linear-error corrected. Note, the signal samples from the second processing channel 134 are demodulated and equalized without the transmission system's receiver filter 124. This is desirable due to the non-commutative operation between a linear function and a nonlinear function as illustrated in FIG. 1. The receiver filter 124 changes the spectrum of an in-band signal, which is the main body carrying the nonlinear behavior of a transmitter. Therefore, the nonlinear behavior of the transmitter estimated after the transmission system's receiver filter is different from that before the receiver filter, which is the true representation of transmitter nonlinearity. The compensated, time-aligned, scaled, equalized signal samples from the second processing channel 134 are output from the demodulator 34 as unfiltered signal samples at four samples per symbol time (Tsymb/4) with in-phase (I) and quadrature (Q) components denoted as {r_i(n), r_q(n)}. Alternatively, the compensated time-aligned, scaled, equalized signal samples may be passed through a down sampler 143 to produce unfiltered signal samples at two samples per symbol time (Tsymb/2) or samples at symbol time (Tsymb). The above described improved demodulator 34 is described in greater detail in co-pending patent application Ser. No. 09/185,419, filed Nov. 3, 1998.

[0031] Referring to FIG. 5 there is shown a representative block diagram of a system incorporating the present invention for in-service measurement of transmitter nonlinearities. The demodulator 34 produces the filtered and unfiltered signal samples that are stored in memory 20. The filtered signal samples have been processed to include filtering through a transmission system receiver filter and the unfiltered signal samples have been processed without passing the samples through the transmission system receiver filter. The unfiltered signal samples, stored in memory 20, are input to a reference signal sample generator 52 via a low pass filter 50. The low pass filter 50, which has a flat amplitude and linear phase response across the signal band, assures the removal of the sum terms in the spectrum that are produced when down converting the IF signal samples to baseband. The unfiltered signal samples are also input to an nonlinear error detector 54 via a delay 56. The delay 56 provides time alignment of the unfiltered signal samples with reference signal samples produced by the reference signal sample generator 52. In an alternative implementation, the filtered signal samples from memory 20 are input to the reference signal sample generator 52 with the unfiltered signal samples being filtered by the low pass filter 50 and the coupled to the nonlinear error detector 54 via the delay 56. The reference signal sample generator 52 produces reference signal samples representing the estimated symbol values that were transmitted at the transmitter of the transmission system with the estimated symbol values in the same state as the signal samples at the output of lowpass filter 50. The reference signal samples are coupled to a magnitude computer 58 and to the nonlinear error detector 54. The magnitude computer 58 produces magnitude values M(n) of the reference signal samples. The reference magnitude values M(n) are coupled to the nonlinear error detector 54. The nonlinear error detector 54 is coupled to the outlier remover 60 which is in turn coupled to the nonlinear error relationship estimator 62. The nonlinear error relationship estimator 62 is coupled to the nonlinear function evaluator 64. The output of the nonlinear function evaluator 64 is coupled to the display 26.

[0032] The reference signal sample generator 52 is illustrated in greater detail in FIG. 6. In the preferred embodiment, the unfiltered signal samples from the demodulator 34 are coupled to the reference signal sample generator 52 via the low pass filter 50. The unfiltered signal samples are first filtered by the transmission system's receiver filter 68 (a complex root raised cosine (RRC) filter in the preferred embodiment) and down sampled to symbol times by down sampler 70 before a multi-region dynamic slicer 72 determines the transmitted symbols. The multi-region dynamic slicer 72 produces signal samples representing symbols denoted as {x_i(n), x_q(n)} from the unfiltered signal samples that have passed through the RRC filter 68. The multi-region dynamic slicer 72 uses symbols {x_i(n), x_q(n)} and slices on x_i(n) to make decisions that estimate the transmitted symbols prior to modulating the transmitter carrier. Since large nonlinear distortion may cause constellation curvature, to perform an accurate symbol estimate, the constellation space having I and Q-dimensions is divided by the multi-region slicer 72 into several regions along the quadrature-dimension (Q-dimension) as shown in FIG. 7. Note that in the preferred embodiment where the unfiltered signal samples are applied to the multi-region slicer 72, the I-phase levels in the multi-region slicer 72 contain the 1.25 pilot level. In the embodiment where the filtered signal samples without the pilot level are applied to the slicer 72, the pilot level needs to be added back-in to generate reference signal samples corresponding to the unfiltered signal samples.

[0033] For each Q-dimension region {R_h, R_m, R_l}, I-dimension reference levels are dynamically estimated from the in-phase I-dimension samples. In the preferred embodiment, the constellation space is divided along the Q-dimension into 3 regions, {R_h, R_m, R_l} as shown in the figure.

[0034] {R_h|x_q(n)>[L(8)−L(1)]/2}  (1)

[0035] {R_m|−[L(8)−L(1)]/2≦x_q(n)≦[L(8)−L(1) ]/2}  (2)

[0036] {R_l|x_q(n)<−[L(8)−L(1)]/2}  (3)

[0037] where L(1), L(2), . . . , L(8) are the ideal constellation levels along the I-dimension without the pilot level (Note: Equations 1, 2 & 3 assume the Q-channel gain is equal to the I-channel gain). From samples {x_i(n), x_q(n)}, the I-dimension constellation level sets L_h(k), L_m(k) and L₁(k) (k=1, 2, . . . , 8) are estimated. In the preferred embodiment of the invention, a Lloyd_Max quantizer is used for clustering the I-dimension values within each region to estimate I-dimension reference levels for each of the regions. The quantizer minimizes the mean square error for a given number of quantization levels or clusters M. For each region, the mean square clustering error is defined as follows: $\begin{array}{cc}\begin{array}{c}\epsilon =E\left[{\left(x-x^{*}\right)}^2\right]\\ =\sum _{k=1}^M{\int }_{t\left(k\right)}^{t\left(k\right)+1}{\left(x-L_x\left(k\right)\right)}^2{p}_x\left(x\right)dx\end{array}& \left(4\right)\end{array}$

[0038] where x is the I-dimension data with probability density function p_x(x),L_x(.)'s are the cluster values to be estimated, and t(k) is the decision boundary between region k−1 and k. Setting the derivatives of ε with respect to t(k) and L_x(k) to zero will give the optimal t(k) and L_x(k). $\begin{array}{cc}\frac{\partial \epsilon }{\partial t\left(k\right)}=0\frac{\partial \epsilon }{\partial L_x\left(k\right)}=0& \left(5\right)\\ t\left(k\right)=\frac{\left(L_x\left(k\right)+L_x\left(k-1\right)\right)}{2}& \left(6\right)\\ L_x\left(k\right)=\frac{{\int }_{t\left(k\right)}^{t\left(k+1\right)}{\mathrm{xp}}_x\left(x\right)dx}{{\int }_{t\left(k\right)}^{t\left(k+1\right)}{p}_x\left(x\right)dx}& \left(7\right)\end{array}$

[0039] Equation (7) show that the optimal quantization level L_x(k) in the decision region [t(k), t(k+1)] is its conditional mean in that region.

[0040] In implementation, the optimal L_x(k)'s are searched using the following iterative clustering process:

[0041] Quantize x_i(n) to A level (A>>M): $\begin{array}{cc}x\left(n\right)=\mathrm{round}\left(\frac{x_i\left(n\right)-x_{\min }}{x_{\max }-x_{\min }}\left(A-1\right)+1\right)& \left(8\right)\end{array}$

[0042] Approximate the probability density function of x(n) by a normalized histogram, denoted as p_x(k) (k=1, . . . , A)

[0043] Initialize decision boundary levels t(k+1) (k=1, . . . , M) evenly across

[0044] A data levels $\begin{array}{cc}L\left(k+1\right)=\mathrm{round}\left(\frac{\mathrm{kA}}{M}\right)+1t\left(1\right)=1& \left(9\right)\end{array}$

[0045] Calculate M clustering levels L_x(k) according to equations (10) and (11)below $\begin{array}{cc}{L}_x\left(k\right)=\frac{\sum _{t=t\left(k\right)}^{t\left(k+1\right)-1}{\mathrm{ip}}_x\left(i\right)}{\sum _{i=t\left(k\right)}^{t\left(k+1\right)-1}{p}_x\left(i\right)}& \left(10\right)\\ t\left(k\right)=\frac{L_x\left(k\right)+L_x\left(k-1\right)}{2}k>1t\left(1\right)=1;t\left(M+1\right)=A+1& \left(11\right)\\ \mathrm{quantization}\mathrm{error}:\epsilon \left(k\right)=\frac{\sum _{t=t\left(k\right)}^{t\left(k+1\right)-1}{\left(i-L_x\left(k\right)\right)}^2{p}_x\left(i\right)}{\sum _{t=t\left(k\right)}^{t\left(k+1\right)-1}{p}_x\left(i\right)}& \left(12\right)\end{array}$

[0046] If a particular region has no elements, i.e. p_x(i)=0 for i=L(k) to L (k+1)−1, then merge this region to its adjacent region. At the end of one iteration (i.e., after k=1 to M), search for a region m which has the maximum quantization error ε(m) and insert an additional decision boundary level at L_x(m).

[0047] Iterate two previous procedures for T times (e.g., T=5)

[0048] Restore the cluster levels L_x(k)'s to the original scale of data and calculate final quantization level L_x(k) (k=1, 2, . . . , M) as follows

[0049] L_x(k)=E[x|(x−L_x(k))²<(x−L_x(j))², ∀j≠k,1≦j≦M]  (13)

[0050] While the above described Lloyd_Max quantizer is implemented in the preferred embodiment of the invention, it is used as an example only. Other clustering methods and search schemes could be used without deviating from the concept of the segmented dynamic slicing.

[0051] The transmitted symbols denoted as s(n) are estimated using a 3-region scheme in the following manner. For example, if x_q(n) ε R_h, L_h(.) is used as the quantizing level set then the closest I-dimension cluster level L_h(m) to x_i(n) is found. The transmitted symbol is then determined as s(n)=L(m). The same procedure applies for x_q(n) ε R_mand x_q(n) ε R_l. The constellation space divided by the multi-region dynamic slicer 72 is used by example only and other constellation space dividing schemes may be used without departing from the scope of the invention as set forth in the appended claims.

[0052] A reference version of the transmitted signal denoted as {c_i(n), c_q(n)} is then regenerated from the estimated transmitted symbols s(n). The reference signal samples are regenerated for comparison by up-sampling the estimated transmitted symbols s(n) to the unfiltered signal sample rate in up-sampler 73 and filtered using a transmission system transmitter filter 74 (an 8-VSB complex RRC filter in the preferred embodiment). The reference signal samples are input to the nonlinear error detector 54 and the magnitude computer 58.

[0053] Referring to FIGS. 8 and 9, the magnitude computer 58 generates reference magnitude values from the received reference signal samples. The reference magnitude values are provided to the nonlinear error detector 54. The nonlinear error detector 54 generates nonlinear error values representing the magnitude error, as shown in FIG. 8, or the phase error, as shown in FIG. 9. For magnitude error detection, magnitude values M_r(n) of the unfiltered signal samples are generated as shown by block 55. There are two different ways of defining the magnitude error. One way is to define the magnitude error as the magnitude difference between reference magnitude values M(n) and the unfiltered magnitude values M_r(n) values:

[0054] magnitude error: E(n)=M_r(n)−M(n)  (14)

[0055] where M_r(n)={square root over (r_i+L (n+L )²+r_q+L (n+L )²+L )}  (15)

[0056] M(n)={square root over (c_i+L (n+L )²+c_q+L (n+L )²+L )}  (16)

[0057] The alternative definition, which is used in the preferred embodiment, defines the magnitude error as the ratio between the unfiltered magnitude values M_r(n) and the reference magnitude values M(n): $\begin{array}{cc}\mathrm{magnitude}\mathrm{error}:E\left(n\right)=\frac{M_r\left(n\right)}{M\left(n\right)}& \left(17\right)\end{array}$

[0058] For phase error detection, as shown in FIG. 9, the phase error is the phase difference between the reference vector {c_i(n), c_q(n)} and the unfiltered vector {r_i(n), r_q(n)} as represented below: $\begin{array}{cc}E\left(n\right)=\measuredangle \left[r_i\left(n\right)+{\mathrm{jr}}_q\left(n\right)\right]\left[c_1\left(n\right)-{\mathrm{jc}}_q\left(n\right)\right]& \left(18\right)\end{array}$

[0059] The nonlinear error detector 54 also arranges the error values as a function of the reference magnitude values.

[0060] Due to imperfect down conversion, reference signal generation error and additive noise in the system, even for small errors, samples with a small magnitude tend to have large errors and therefore large error variations, which affect the accuracy of the measurement. To overcome this problem, the following is performed by applying the nonlinear error values derived from the reference and unfiltered signal samples in the nonlinear error detector 54 to the outlier remover 60. The outlier remover 60 establishes boundaries or thresholds in nonlinear error versus magnitude space for removing nonlinear error values that affect the accuracy of the measurement. The signal magnitude range, derived from the reference signal samples, is divided into several subranges R_M(k) (k=0, 1, . . . K) in the outlier remover 60. $\begin{array}{cc}{M}_1=\underset{n}{\min }\left\{M\left(n\right)\right\}& \left(19\right)\\ M_2=\underset{n}{\max }\left\{M\left(n\right)\right\}& \left(20\right)\\ R_M\left(k\right)=M_1+k{M}_2-\frac{M_1}{K}& \left(21\right)\end{array}$

[0061] In each subrange, R_M(k)≦M(n)<R_M(k+1), those nonlinear error values having large deviations are dropped from the nonlinear function estimation. For small magnitude regions, (e.g. k≦K/3), the decision boundary is determined based on a probability histogram and a prior p(k). As shown below, the boundary is determined so that within the boundary the data probability would be equal to p(k) and the probability density should be the largest in the subrange. $\begin{array}{cc}P(E_{\mathrm{low}}\left(k\right)\le E\le E_{\mathrm{up}}\left(k\right)=p\left(k\right)& \left(22\right)\\ \left\{E_{\mathrm{low}}\left(k\right),E_{\mathrm{up}}\left(k\right)\right\}=\arg \max \left\{\frac{p\left(k\right)}{E_{\mathrm{up}}\left(k\right)-E_{\mathrm{low}}\left(k\right)}\right\},\forall E\in R_M\left(k\right)& \left(23\right)\end{array}$

[0062] The upper and lower boundaries are linearly interpolated between regions to have a smooth boundary contour.

[0063] While the above described histogram-based method is applicable for large magnitude regions, it proves to be computationally inefficient. The decision boundaries or thresholds for detecting outliers in the large magnitude regions may be implemented in the following manner for large magnitude regions. The decision boundaries are based on the mean and variance of the nonlinear error values in the respective large magnitude subranges.

[0064] mean of the error values in subrange k:

[0065] m_k=E[E(n)|R_M(k)≦M(n)<R_M(k+1)]  (24)

[0066] variance of the error values in subrange k:

[0067] σ_k²=E[(E(n)−m_k)²|R_M(k)≦M(n)<R_M(k+1)]  (25)

[0068] k=0, 1, . . . K−1

[0069] Threshold TH_k=ασke.g.,=α=1  (26)

[0070] vector {E(i),M(i)}={E(n),M(n)| for |E(n)−m_k|≦TH_k}  (27)

[0071] Magnitude and phase nonlinearities are defined as a function of instantaneous signal magnitude. Therefore, to estimate a nonlinear function, the error set {E(n)} is ordered as a function of signal magnitude M(n) in the nonlinear error relationship estimator 62 and a least square based (LS) polynomial curve fitting, although other regression methods may be used, is performed on the vector array {E(n), M(n)}. A polynomial curve fitting has the following form:

[0072] y=a₀+a₁x+a₂x²+. . . +a_Dx^D  (28)

[0073] where D is the polynomial order. A least square magnitude nonlinearity curve is then derived based on the following equations. Due to the fact that large nonlinear error often occur when the signal has a large magnitude, and the 8-VSB signal has a small probability of large magnitudes, a weighting function is applied to the LS curve fitting procedure to emphasize the large magnitude data pair. For example, for magnitude data greater than 0.75 (L₍₈₎−L₍₁₎), multiply both sides of the curve fitting function by a factor of 3. Also, to improve estimation accuracy, the reference magnitude values entering the least square curve fitting are shifted by a constant value to centerize the magnitude values (e.g., M(n)=M(n)−mean (M(n)). $\begin{array}{cc}E=\mathrm{MA}& \left(29\right)\\ E={\left(\left[E\left(1\right),E\left(2\right),\dots ,E\left(N\right)\right]\right)}^T& \left(30\right)\\ A={\left(\left[a_0,a_1,a_2,\dots ,a_D\right]\right)}^T& \left(31\right)\\ M=\left[\begin{array}{cccc}1& M\left(1\right)& \cdots & {M\left(1\right)}^D\\ 1& M\left(2\right)& \cdots & {M\left(2\right)}^D\\ ⋮& ⋮& \cdots & ⋮\\ 1& M\left(N\right)& \cdots & {M\left(N\right)}^D\end{array}\right]& \left(32\right)\\ A={\left(M^{T}M\right)}^{-1}{M}^{T}E& \left(33\right)\end{array}$

[0074] From the estimated polynomial coefficients A, a smoothed polynomial function is derived over the range of signal magnitude.

[0075] E(M)=a₀+a₁M+. . . +a_DM^DMε signal magnitude range  (34)

[0076] For the magnitude errors as defined in equation (14), the gain of a transmitter amplifier, or nonlinear error values of the magnitude, is then calculated as a function of signal magnitude in the nonlinear function evaluator 64 from the following expression. $\begin{array}{cc}{G}_{\mathrm{dB}}\left(M\right)=20{\log }_{10}\left(\frac{E\left(M\right)}{M}+1\right)\left(\mathrm{dB}\right)& \left(35\right)\end{array}$

[0077] For magnitude errors as defined in equation (17), the gain of the transmitter amplifier, or nonlinear error values of the magnitude, is then calculated as a function of signal magnitude in the nonlinear function evaluator 64 as: $\begin{array}{cc}{G}_{\mathrm{dB}}\left(M\right)=20{\log }_{10}\frac{E\left(M\right)}{a_0}& \left(36\right)\end{array}$

[0078] For nonlinear phase errors, the nonlinear error values of the phase are given by equation 34 as:

[0079] θ(M)=a₀+a₁M+. . . +a_DM^DMε signal magnitude range  (37)

[0080] FIGS. 10 and 11 respectively show representations of graphical displays of the nonlinear magnitude values versus a range of reference magnitude values and the nonlinear phase error values versus the range of reference magnitude values.

[0081] The above system for in-service measurements of transmitter nonlinearities has been described as a series of instruction executed by a processor of a transmission system receiver on signal samples stored in a memory. The elements and structure of the in-service transmitter nonlinearity measurement system and method of the present invention may equally be implemented using hardware circuitry. The elements of the system may implemented as discrete filters, logic circuits and delay lines. Alternatively, the elements may be implemented in one or more application specific integrated circuit (ASIC) device(s) or a combination or discrete circuit element and ASIC device(s).

[0082] A system for in-service transmitter nonlinearity measurements has been described that measures such nonlinearities by way of comparing generated reference signal samples to linear error-corrected unfiltered signal samples in order to measure the nonlinear error. Linear distortion is removed from the received signal samples and the linear error-corrected unfiltered signal samples are computed from the received signal samples without applying the transmission system's receiver filter. The reference signal samples are then estimated and regenerated from the unfiltered signal samples or filter signal samples where the transmission system's receiver filter is applied. No training sequence is required. A multi-region dynamic slicer accurately estimates the reference signal samples. In comparing the unfiltered signal samples with the reference signal samples, nonlinear distortions are distinguished from other noise-like distortions by utilizing the systematic nature of transmitter nonlinearity and the random nature of other distortions by computing a weighted, least-square based polynomial regression on the nonlinear error samples estimated from the comparison of the generated reference signal samples and the unfiltered signal samples.