Optical error simulation system

[0001] This application claims priority under 35 U.S.C. 119 to prior Australian Patent Application No. PR8189, filed Oct. 10, 2001, the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the estimation of signal quality in simulation of optical communications systems.

[0003] Recently, there has been an explosion in the use of optical fibre telecommunications as the main backbone for the carriage of data in telecommunications networks.

[0004] The installation or maintenance of large complex optical networks is often a major and expensive undertaking. Obviously, in planning, certain capacity requirements may have to be met. Optical systems operate in the real world and are subject to error rates due to inherent noise in the equipment utilized in the system and in nature generally.

[0005] In the planning of large scale optical systems, it is desirable to be able to simulate the performance of a system before installation or change so as to determine whether operational requirements will be met. Ideally, such simulation systems also provide for the simulation of bit error rates when applied to the system.

[0006] Unfortunately, the methods previously utilized in the simulation of bit error rates often lead to inaccurate simulation results. This is often a result of utilizing perhaps inappropriate Gaussian noise statistics when simulating bit error rates of an optical system.

[0007] It is an object of the present invention to provide for an improved method and system for the simulation of optical errors.

[0008] The invention resides in the recognition that the random noise processes that affect an optical signal during transmission may be modeled independently of the noise processes that affect the signal after conversion to the electrical domain in the receiver. The statistical properties of the received optical signal may be analyzed initially without including further random fluctuation caused by post-detection noise. Then, given a set of parameters characterizing the post-detection noise processes, a measure of quality such as the bit error rate may be calculated including the effect of the pre-detection and post-detection noise processes in combination.

[0009] One aspect of the present invention provides a method for estimating a measure of the quality of a received signal in a computer simulation of an optical transmission system, wherein the simulation includes additive optical noise generated by components within the transmission system. The method includes the steps of: (a) calculating one or more sets of parameters of a first non-gaussian probability density function so as to approximate a suitable fit to the amplitude distribution of the received signal, including the effects of deterministic processing in the receiver but excluding the effects of statistical noise fluctuations introduced by the receiver; (b) calculating one or more sets of parameters of a second non-gaussian probability density function that approximates a suitable fit to the amplitude distribution of the received signal, including the effects of deterministic processing and statistical noise fluctuations in the received signal using the calculated set(s) of parameters of the first non-gaussian probability density function and pre-specified statistical properties of the noise fluctuations introduced by the receiver; and (c) computing the measure of quality using the set(s) of parameters of the second non-gaussian probability density function.

[0010] The phrase deterministic processing refers to the operations performed within the receiver that are fully determined by the characteristics of the receiver components and which produce results without random variation, such as the process of low-pass filtering of the signal which may be performed prior to sampling of the received signal.

[0011] The phrase statistical noise fluctuations refers to the processes which occur in the receiver which are random in nature and which produce results that are subject to statistical variation, such as the electrical noise processes which naturally occur in components such as resistors and other electronic devices.

[0012] The method of the invention provides a number of advantages over prior art techniques, such as the ability to obtain more accurate estimation of the measure of signal quality (e.g. bit-error rate) over a simulation run of a given duration or, alternatively, an equally accurate estimate over simulation run of substantially reduced duration.

[0013] In one embodiment the components of the transmission system include optical amplifiers. The optical amplifiers may comprise erbium-doped fiber amplifiers. The statistical properties of the noise introduced by the receiver may comprise a Gaussian distribution characterized by its mean and variance.

[0014] Preferably, the first non-gaussian probability density function comprises a non-central x² probability density function. The x² probability density function is known to be the exact form of the distribution of the detected signal amplitude in the case of ideal reception of an optical signal corrupted by additive noise such as that introduced within optical amplifiers. It has been discovered by the present inventors that this distribution is a good fit in the case of many practical systems designed to approximate as closely as possible the ideal system.

[0015] Another aspect of the present invention provides a computer program embodied on a computer readable medium for estimating a measure of the quality of a received signal according to the above described method.

[0016] A further aspect of the present invention provides an apparatus for implementing the above described method.

[0017] In a preferred embodiment the received signal comprises an intensity modulated digital signal and the measure of the quality of the received signal may comprise the bit error rate. At least two sets of parameters for each of the first and second non-gaussian probability density functions may be calculated, corresponding respectively to the amplitude distributions of received data ones and received data zeros.

[0018] Preferably, the received signal is passed through a narrow optical filter prior to reception.

[0019] Preferred and other embodiments of the present invention will now be described by way of illustration only with reference to the accompanying drawings in which:

[0020] FIG. 1 illustrates schematically an example simulation system to which the method of the preferred embodiment can be applied;

[0021] FIG. 2 is a graph showing the variance of Bit Error Rates with received optical power;

[0022] FIG. 3 is a graph showing a comparison of back to back BER curves for different estimation methods;

[0023] FIG. 4 is a graph showing system BER curves for different estimation methods for an optical filter bandwidth of 80 GHz;

[0024] FIG. 5 is a graph showing the optimum threshold setting as a function of received optical power for four different estimation methods;

[0025] FIG. 6 is a graph showing the comparison of system BER curves for a hybrid Gaussian approximation and hybrid chi-squared method using a relative threshold setting of 0.32;

[0026] FIG. 7 is a graph showing the comparison of system BER curves for a hybrid Gaussian approximation and hybrid chi-squared method using a relative threshold setting of 0.18;

[0027] FIG. 8 is a graph showing a comparison of system BER curves for the different methods for an optical filter bandwidth of 25 GHz;

[0028] FIG. 9 is a flow chart of the steps in the preferred embodiment; and

[0029] FIG. 10 illustrates the sampling process utilized in the preferred embodiment.

[0030] In the preferred embodiment, there is provided a method for estimating the Bit Error Rate (BER) of a received signal in a computer simulation of an optical transmission system, including the effects of post-detection degradation such as electrical noise and photodetector dark current.

[0031] The preferred method includes the steps of calculating the parameters of non-central x² probability density functions that are best-fits to the distributions of the amplitude of the received signal in the absence of post-detection degradation effects and computing the BER of the received signal, including the effects of post-detection degradation effects, using the calculated parameters of the non-central x² probability density functions, and the parameters of the post-detection degradation effects.

[0032] By way of explanation of aspects of the method of the preferred embodiment, the operation of the method is described in an example simulation environment. An example system for use with the method is illustrated in FIG. 1. Such a system may be simulated in software operating on standard computer hardware. In the example 1, the optical data source 2 is considered to be a single frequency laser modulated by a pseudo-random bit sequence 3 (PRBS) with a nominal 50% mark ratio (i.e., equal numbers of ones and zeros generated on average). Optical white noise 4 is added 5 to the signal, which has a zero-mean Gaussian field amplitude distribution. This may represent additive amplified spontaneous emission (ASE) noise generated in optical amplifiers, for example. The signal and noise are filtered using an optical bandpass filter 6 centered on the output frequency of the laser. The combined optical signal and noise are received by direct-detection with an ideal PIN photodetector 8, and then additional noise 9 is introduced 10 representing the electrical noise added in the components of the receiver circuitry. Low pass filtering is applied 11 to the resulting signal, and the output of the filter is sampled 12 at a frequency equal to the bit rate. The sample values represent the signal amplitude (voltage or current) at the sampling instants, and the distribution of these values can be used to estimate the overall probability density functions (PDFs) for transmitted data ones and zeros. These PDF estimates are then used to calculate estimates of the BER at the receiver 14.

[0033] Now different methods can by used to simulate the presence of noise 9 in the receiver and hence for determining the BER of the system 1. Example methods can include:

[0034] 1. A prior art method comprising an ideal system model using an integrate-and-dump receiver.

[0035] 2. A prior art method comprising a pure stochastic sampled model using Gaussian approximation.

[0036] 3. A method according to a preferred embodiment of the present invention comprising a stochastic sampled model assuming chi-squared statistics for the received optical noise, combined with semi-analytical calculation of the impact of post-detection noise (a hybrid chi-squared method).

[0037] 4. A prior art method comprising a stochastic sampled model assuming Gaussian statistics for the received optical noise, combined with semi-analytical calculation of the impact of post-detection noise (a hybrid Gaussian method).

[0038] Taking each model separately:

[0039] 1. Ideal System with Integrate and Dump Receiver (Gauss I&D)

[0040] A calculation method has previously been described for an ideal system with an integrate and dump (I&D) receiver in D. Marcuse, Calculation of Bit-Error Probability for a Lightwave System with Optical Amplifiers and Post-Detection Gaussian Noise, J Lightwave Technol., 9 (4), April 1991, pp. 505-513), the entire content of which is incorporated herein by cross reference. In this case, a number of assumptions are made, including the assumption that all energy in each bit is confined to its corresponding bit interval. The optical band pass filter is a perfect rectangular filter with a bandwidth of B_opt. The electrical low pass filter consists of an ideal integrator which integrates over the full bit period. The sampling instant occurs at the end of the integration time, and the integrator then instantaneously resets (i.e., it is memoryless).

[0041] In this particular case, analytical results for the PDFs and the BER can be obtained under a Gaussian approximation. In the case where the post-detection Gaussian noise is negligible, the statistics of the detected optical field follow a chi-squared distribution. When both the optical noise and post-detection Gaussian noise must be considered, the resulting distribution can in principle be obtained from the convolution of the appropriate chi-squared and Gaussian PDFs, however there is no known closed-form expression for the resulting distribution function. Efficient numerical methods have been developed by Marcuse to calculate the PDFs and corresponding error probabilities, which are highly accurate for low error rates (less than 10³) that are of the greatest interest. This numerical calculation method shall be referred to as the Marcuse method.

[0042] Under the assumptions above, the system is highly idealized, and the results obtained are often optimistic compared to a realistic implementation. The I&D receiver is similar in performance to an ideal filter and sample receiver with a perfect matched filter with noise equivalent electrical bandwidth B_e1/(2T) (where T is the bit period). Such systems are not realized in practice at the very high bit-rates typically employed in baseband optical transmission systems. Thus the results from this ideal system model are likely to indicate artificially superior performance as compared to those obtained from a model constructed from realistic individual component models.

[0043] Furthermore, the idealized I&D system model makes assumptions about the entire system, not just the receiver components and BER estimation. Therefore it cannot be applied to the simulation of more general systems comprising further components in addition to the idealized transmitter, receiver and noise sources represented in FIG. 1.

[0044] 2. Stochastic Sampled Model: Gaussian Approximation

[0045] In the absence of any more specific knowledge of the expected form of the PDF, the most common approach employed is simply to assume that the statistics are Gaussian. A sampled numerical simulation of the system can be carried out, with the noise sources modeled by using Gaussian-distributed pseudo-random number generators to generate noise samples that are added to the signal samples. At the output of the electrical filter, the mean and variance of the resulting samples can be calculated at each sampling instant across the bit-frame, which are sufficient to fully determine the statistics under the Gaussian approximation. So long as a sufficiently large number of bit periods are simulated (typically greater than 10³ bits), the estimates of mean and variance can have small confidence intervals, and the method is fairly effective.

[0046] The use of a sampled simulation technique enables realistic models of the system components to be used. It also enables general system models to be constructed, using simulation models of all the components that exist in the real system. In the example, the following component models were used. The signal source 2 comprises a single frequency CW laser emitting at an optical frequency of 193.1 THz, modulated via an external Mach-Zehnder modulator with an extinction ratio of 30 dB. The driving signal to the modulator consists of an NRZ waveform derived from the generated PRBS 3, with Gaussian edges having rise and fall times equal to one quarter of the bit period. The optical noise source 4 was implemented as a zero-mean Gaussian pseudo-random generator producing white noise samples resulting in a constant noise power spectral density over the full sampled optical bandwidth. The variance of this generator is expressed as the desired optical noise power spectral density (in units of W/Hz).

[0047] The optical bandpass filter 6 was assumed to be a trapezoidal filter that closely approximates the ideal rectangular filter assumed in the ideal system model. It had a flat-top bandwidth of B_opt, and the transmission fell away rapidly on either side of this bandwidth, reducing to 10 dB within 1 GHz of the pass band edges.

[0048] An ideal PIN photodetector 8 was assumed, with a quantum efficiency of unity, corresponding to a responsivity of 1.252 A/W 193.1 THz. Dark current was not considered in the example, and shot noise was negligible compared to the optical and electrical additive noise, and was excluded from the model.

[0049] The electrical noise source 9 was implemented as a zero-mean Gaussian pseudo-random generator producing white noise samples resulting in a constant noise power spectral density over the full sampled electrical bandwidth. The variance of the generator is expressed as the desired electrical noise current density (in units of A/Hz^1/2). The electrical filter 11 used in the example was a third-order Bessel filter, with bandwidth equal to 70% of the bit rate. The sampling time is taken to be at the center of the bit frame.

[0050] 3. Stochastic Sampled Model: Hybrid Chi-Squared Method (BER Stochastic Chi 2)

[0051] The principal disadvantage of the above Gaussian-approximated stochastic model is that the statistics of the detected optical signal are non-Gaussian. In particular, a square-law detector cannot produce negative output values, and the resulting distributions will therefore exhibit skewness, which is not a feature of the normal distribution. One possible approach would be to estimate a third moment of the sample distributions at the input to the BER estimator, in addition to mean and variance. However this is likely to be very inefficient, since a much larger number of bits must be simulated in order to obtain an acceptably accurate estimate of a higher-order moment. In the method of the preferred embodiment, a specific form for the PDF is assumed, and a fitting method is tailored to the assumed distribution. In general, the action of linear filtering in both the optical and electrical domains alters the PDF observed at the BER estimator. Linear operations performed on Gaussian-distributed random variables result in outputs that are also Gaussian distributed. However, this is not true in general, and so the exact form of the distributions appearing at the BER estimator is not known for an arbitrary system. In the method of the preferred embodiment, a hybrid x²/Gaussian distribution is utilized in the example arrangement.

[0052] Four parameters are required to fully specify the hybrid x²/Gaussian distribution. The x² distribution (non-central, in general) introduces three parameters in its classical form: a scaling factor, the non-centrality parameter and the number of degrees of freedom. A number of alternative parameterizations of the distribution are possible, such as those used in the preferred embodiment of the present invention, however in all such cases there will remain three independent parameters.

[0053] Description of the post-detection Gaussian noise requires an additional fourth parameter, namely the electrical noise variance. Estimation of these four parameters based purely upon the samples input to the BER estimator is infeasible for reasonable sample sizes using a general technique such as the method of moments. Thus an improved technique for use with the preferred embodiment has been developed.

[0054] The first step in the method is to remove the requirement to estimate the electrical noise variance. Instead of constructing the receiver literally as shown in FIG. 1, the electrical noise source 9 is removed so that the samples arriving at the BER estimator 14 comprise only the filtered signal with received optical noise. Estimates of the mean and variance of these input samples are computed, and these estimates used to construct an expression for the non-central x² distribution that depends only upon the number of degrees of freedom. Then a histogram of the received samples is constructed, and the number of degrees of freedom estimated by finding the value that produces a best-fit x² distribution to the histogram data. The final PDFs and BER are calculated using Marcuse's method to include the effect of the total electrical noise variance that can be obtained by multiplying the electrical noise power spectral density by the electrical filter bandwidth. Thus the BER estimator now requires these two quantities to be provided as additional parameters. This implies some loss of generality in the application of the algorithm to the example, since the electrical noise is not passed through the actual processing elements of the receiver. However, the general receiver structure shown in FIG. 1 is the most common structure employed in baseband digital optical transmission systems, and so the impact of this loss of generality is minimal.

[0055] 4. Stochastic Sampled Model: Hybrid Gaussian Method (BER Stochastic Gauss)

[0056] There may be cases in which the x² distribution does not produce a good fit to the histogram constructed from the samples received by the BER estimator. For example, it may be that the assumption of optical noise dominated by additive ASE is invalid due to other sources of noise being present in the system. Or the specific nature of the optical and/or electrical filtering employed may result in a distribution that is not well approximated by a chi-squared distribution with any number of degrees of freedom. In this case, it is desirable for the BER estimator to be able to fall back to a Gaussian approximation, either automatically or under user control. This results in a hybrid Gaussian method, in which the mean and variance of the received optical signal without additive electrical noise are first estimated, and then the additional variance of the post-detection Gaussian noise is added to the estimated variance.

[0057] Testing Method Examples

[0058] Comparative tests were performed by calculating complete BER curves, rather than individual BER values. A BER curve can be obtained by varying the total optical power at the input to the receiver 8, and measuring the BER as a function of this received optical power. Experimentally, such curves can be obtained by placing a variable attenuator in front of the receiver. In the simulation tests, the same effect is achieved more directly by placing a power controller in front of the receiver. The power controller comprises a numerical module that scales the input samples to the desired average power representation. As the received power is varied, the optical noise 4 varies in proportion, however the post-detection electrical noise 9 is constant. Thus at relatively low levels of received optical power, the receiver noise dominates, and the PDFs at the BER estimator input have strongly Gaussian characteristics. At high levels of received optical power, the optical noise may dominate, and the PDFs at the BER estimator input have strongly non-Gaussian characteristics. When optical noise is present there will be a high-power limit to the minimum BER achievable, corresponding to the effect of the optical noise alone. This is usually known as the BER floor.

[0059] When there is no optical noise present in the received signal, the BER measurement will characterize the performance of the receiver (and, to a lesser extent, the transmitter). The BER will be the result of the additive electrical noise 9 only, and as the received optical power is increased, the signal-to-noise ratio will increase, and thus the BER will decrease indefinitely. For purely Gaussian noise, the resulting curve is straight line on the commonly used error function scale. This line represents the limits of the receiver performance, and can be referred to as a back-to-back BER curve.

[0060] To fully characterize a given optical system, it is usual to measure both the back-to-back curve (i.e., the receiver limit) by connecting the transmitter directly to the receiver, followed by a system curve in which the optical system is included. FIG. 2 illustrates the back to back measurement 20 and a typical system measurement 21 . The difference between the two curves 20, 21 represents the system impairment.

[0061] For the example system of FIG. 1, back-to-back BER curves can be obtained by simply turning off the optical noise source (i.e., by setting its power spectral density to zero). In order to allow for a fair comparison between the four different methods, the electrical noise power was set differently for the ideal I&D model, and the three sampled simulation models, such that in all cases the receiver sensitivity was 20 dBm for a BER of 10⁹. In order to achieve this, the electrical noise power is lower for the sampled simulation model than for the ideal I&D model, in order to compensate for the higher effective bandwidth of the low-pass filter compared to the ideal integrating filter. Note that the ratio ²_th(ID)/²_th(s)1.42, i.e., almost exactly the ratio of the simulation filter bandwidth to the effective bandwidth of the integrate and dump filter (0.7/0.5). The system and receiver parameters for the examples are shown in Table 1.

TABLE 1
General system and receiver parameters for tests.
		Value for
Parameter Symbol, Units	Value for I & D	Simulation
and Description	Model	Models
BR (Gb/s) Bit rate	10
Bopt (GHz) Optical filter	80
bandwidth
th (A/Hz1/2) Electrical noise	2.947 1011	2.466 1011
current
Be (GHz) Electrical filter	5.0 (i.e., BR/2 effective	7.0
bandwidth	noise equivalent
	bandwidth)
Nb Number of transmitted bits		8192

[0062] The results of the back-to-back tests are shown in FIG. 3. The following features are of interest. The Gaussian approximation method, which uses a purely statistical estimate of the mean and variance on each run, produces essentially the same results as all other methods. Some variability is apparent, due to the fact that on each run a different set of electrical noise samples is generated, resulting in some statistical fluctuations. However, for the number of bits simulated (8192) these fluctuations are small.

[0063] The hybrid gaussian and chi-squared methods also produce the same results as the BER visualizer and the I&D model. This confirms the validity of applying the electrical noise density and bandwidth as parameters to the BER estimator. In the BER visualizer, the electrical noise has been generated and then passed through the filter, whereas in the case of the two hybrid methods the noise variance is computed directly from the known noise spectral density and filter bandwidth. The results are, for all practical purposes, identical. The hybrid methods exhibit no statistical fluctuations, because the effect of the electrical noise is computed deterministically.

[0064] For the system measurements, additive optical noise was considered that was fully polarized parallel to the signal polarization state. For all methods, the total additive optical noise power spectral density was set so that the optical signal-to-noise ratio, measured in the optical frequency domain using a 0.1 nm (12.479 GHz 1550 nm) resolution bandwidth, was 18 dB. (Note that since the total received optical power is set at the input to the receiver, the absolute value of additive optical noise is not relevant, however for the simulation methods, the absolute values of signal power and noise power spectral density were 1.01 mW and 2.5610¹⁵ W/Hz respectively at the input of the power combiner).

[0065] The results of the system BER curves for the different methods for an optical filter bandwidth of 80 GHz are shown in FIG. 4. A number of observations can be made.

[0066] The I&D models predict significantly better performance than the simulation models. As previously noted, this may be expected since they represent an ideal system that will never be achieved in practice. In fact, the difference could be explained by the difference in effective electrical bandwidth of the two receiver models, as follows. For the Gaussian approximation, the relationship between the Q-factor and the BER is given by $\begin{array}{cc}\mathrm{BER}=\frac{1}{2}\mathrm{erfc}\left(\frac{Q}{\sqrt{2}}\right)\frac{1}{\sqrt{2}Q}{}^{-\frac{Q^2}{2}}& \left(1\right)\end{array}$

[0067] Furthermore, if the signal-to-noise ratio is reasonably high, the relationship between the Q-factors Q₁ and Q₂ of two systems having electrical noise bandwidths of B_n1 and B_n2 respectively, is given by $\begin{array}{cc}\frac{Q_2}{Q_1}\sqrt{\frac{B_{\mathrm{n1}}}{B_{\mathrm{n2}}}}& \left(2\right)\end{array}$

[0068] Considering the Gaussian approximation results shown in FIG. 4, a BER of 10⁹ is obtained in the simulated system at a received optical power of about 12.2 dBm. The corresponding Q-factor is 6.0. Thus, the predicted Q-factor and BER for the I&D system using Equations (1) and (2) are Q6.0{square root}{square root over (0.7/0.5)}7.1, and BER6.410 ¹³. This BER value corresponds very closely to the value calculated using the Gaussian approximation for the I&D system.

[0069] For low received optical power, where post-detection Gaussian noise dominates, all methods produce similar results. This is to be expected, given that the back-to-back measurements in which the noise was purely Gaussian, were identical. The offset between the back-to-back curve and the system curves at low received power is due to the effective extinction ratio degradation caused by the average optical noise current (the received optical power is a total, and therefore includes the optical noise power). This is a commonly observed phenomenon in experimental BER measurements.

[0070] The difference between the Gaussian approximation and the improved distribution estimate is similar for both the I&D method, and the sampled simulation methods. In all cases, the Gaussian approximation overestimates the BER. There are also significant differences in the corresponding optimum threshold settings. FIG. 5 shows the optimized threshold settings determined using the I&D model with Gaussian approximation, the I&D model using the Marcuse method, and the hybrid Gaussian and chi-squared simulation models. Although the error probabilities calculated for the idealized I&D system, and the simulated system, are quite different, the corresponding optimum threshold settings are very similar, for both the Gaussian approximation, and the chi-squared case. This reflects the fact that although the low pass filter in the simulated system allows more noise to pass, resulting in a degraded SNR at the BER estimator, the statistical form of the noise in both models is very similar. As in the back-to-back results, the BER visualizer produces very similar results to the hybrid method using the Gaussian approximation at all optical power levels. This further confirms that for the standard baseband direct detection receiver consisting of a PIN detector and a low pass filter, there is no appreciable difference between the results obtained by passing random noise samples through the low pass filter model, and those obtained by computing the effect of post-detection noise directly from the noise spectral density and the filter bandwidth.

[0071] The observation above regarding the differing optimum threshold for the Gaussian approximation and the more accurate methods is significant. Comparison of the final, optimized BER estimates alone suggests that the difference between using a Gaussian approximation and a more detailed model of the noise statistics, is relatively small. However, the corresponding optimum threshold settings are quite different, for both the ideal I&D system model, and the simulated system model. The implication is that the Gaussian approximation can result in the calculation of an incorrect optimum threshold setting, however because the (incorrect) Gaussian assumption is then also used to calculate the BER, the consequences of the error are reduced.

[0072] All the previous results used an optimized decision threshold setting when calculating the final BER estimates. In a practical system, true optimization of the threshold is rarely performed. Instead, receiver filters with a finite low frequency cutoff are generally used, with automatic gain control or equivalent circuitry to allow a threshold to be set relative to the received signal voltage or current amplitude. Thus, for many practical cases, a fixed relative threshold setting is a more realistic model of this receiver circuitry.

[0073] FIG. 5 suggests that quite different BER estimates will be produced for the Gaussian and hybrid chi-squared estimation methods. In particular, if it is desired to simulate a system using a relative threshold setting, then an accurate model of the noise statistics is desirable if the results are to be comparable to those that would be obtained by experimental measurement of a real system.

[0074] FIG. 6 shows the results of a simulation using a relative threshold setting of 0.32 (i.e., the threshold is set to a point 32% of the way between the mean zero and one levels), using both the hybrid Gaussian and hybrid chi-squared estimation methods. The OSNR has been increased to 20 dB for this simulation (by decreasing the optical noise power spectral density to 1.6210¹⁵ W/Hz), to allow for the fact that the relative threshold setting results in degraded performance compared to using am optimized decision threshold. Otherwise, all parameters of this simulation are the same as for the previous tests. For a system with additive optical noise, it will always be desirable to set the threshold closer to the zero level, because the variance of the noise on data ones is larger, due to the higher level of signal-spontaneous beat noise.

[0075] FIG. 6 shows that if a simulation is performed using the Gaussian approximation, very poor performance results compared to the hybrid chi-squared methodover four orders of magnitude difference in the error-rate floor. For low received optical power, where the Gaussian post detection noise dominates, the results are very similar. However, once the optical noise starts to become significant, the two BER curves 61, 62 rapidly diverge. The hybrid chi-squared method provides a superior model of the actual noise statistics at the input of the BER estimator. Thus the results using the hybrid chi-squared method provides a more accurate representation of real system performance. At best, the Gaussian approximation can be seen to produce a conservative estimate of the system performance, however with such a significant discrepancy it is likely that the use of the Gaussian approximation will result in significant over-engineering of a resulting system. It is equally possible that a lower relative threshold setting could be chosen, for which the Gaussian approximation results in predictions of superior BER performance. In this case, the real system could perform significantly worse than predicted by the simulation.

[0076] FIG. 7 shows BER curves where the relative threshold setting has been set to 0.18 (closer to the optimum for Gaussian statistics). In this case, a simulation using the Gaussian approximation predicts an error rate floor at 10¹³, whereas the more realistic hybrid chi-squared method predicts an error rate floor at around 10¹². There is therefore a very real risk that if the Gaussian approximation is used to determine a suitable relative threshold setting, the performance of the real system may be inferior to that predicted. This is in contrast to the general view that the Gaussian approximation produces a conservative estimate of system performance.

[0077] When narrow optical filters are employed, such that the ratio of the optical to electrical bandwidth becomes small (on the order of one), the post-detection statistics of the optical noise become less well-approximated by the Gaussian distribution. This is because the number of degrees of freedom in the non-central chi-squared distribution is smaller. (The non-central chi-squared distribution becomes Gaussian in the limit of large number of degrees of freedom.) In addition, the narrow optical filtering will tend to introduce significant correlations into the post-detection noise that result in the I&D model becoming invalid. FIG. 8 shows the results for the sampled simulation methods when the optical filter bandwidth is reduced to 25 GHz. A slight reduction is apparent in the difference between the back-to-back curve and the system curves at low received optical power. This is a direct result of the reduction in average optical noise power/current due to the reduction in filter bandwidth. The difference between the results using a Gaussian approximation, and those using the hybrid chi-squared method is more significant, as expected. The Gaussian approximation once again produces a slightly lower estimate of the BER than the stochastic Gaussian approximation. The error rate floor is reduced by about two orders of magnitude (for the hybrid chi-squared method) compared to the system using an 80 GHz optical filter bandwidth. This is due mainly to the reduced spontaneous-spontaneous beat noise, which is the main contributing noise source for data zeros.

[0078] These examples generally show the advantages of the practical application of the hybrid chi-squared BER estimation method.

[0079] For the example system considered, in the absence of optical noise at the receiver input, very good agreement was obtained between the hybrid method in which post-detection Gaussian noise is included deterministically, and the existing purely statistical method in which electrical noise is added prior to low-pass filtering. The difference between a Gaussian approximation, and the new hybrid chi-squared method is consistent with similar results obtained for a semi-analytical technique based on an idealized I&D system model. The example demonstrates that such an idealized system model significantly over-estimates the performance of a practical system. The improved BER estimation technique allows the improvement in system performance due to the non-gaussian statistics of received optical noise to be predicted. If an optimized decision threshold setting is used, the difference between a Gaussian approximation and the new method is most significant when narrow optical filters are employed, with bandwidth of similar order to the electrical bandwidth. For realistic systems using a non-optimized decision threshold, significant errors in the overall BER can occur if an inaccurate model of the detected noise statistics is used, and it is in this case that the hybrid chi-squared method is most clearly superior to a Gaussian approximation.

[0080] The main disadvantage of the hybrid chi-squared technique is that it is required to estimate an additional parameter from the input samples, and therefore it is somewhat more susceptible to statistical variations resulting from the finite input sequence. As a result, it generally may require a greater number of bits to be simulated in order to compute results with comparable confidence to the Gaussian approximation methods. The Gaussian approximation remains a reliable fall-back when the hybrid chi-squared technique is not applicable, or when a conservative lower bound on system performance is desired.

[0081] Turning now to a more detailed discussion of the theory and algorithm required to implement a hybrid x² system. Initially, it is necessary to define a number of important variables, which in the preferred embodiment are parameters provided by the user.

Symbol	Definition
T	The bit period. Determined by a parameter BitRate
	(T 1/BitRate).
ts	The sample time specified by a parameter SampleTime.
r	The sample range specified by a parameter SampleRange.
Dt	The decision threshold specified by a parameter Threshold.
2p	The post-detection noise power spectral density added in
	the receiver, in current units, as specified by
	a ThermalNoise parameter.
id	Dark current, as specified by a DarkCurrent parameter.
Be	Equivalent electrical noise bandwidth, as specified by a
	ElecNoiseBandwidth parameter.

[0082] The probability of a bit being in error is as follows:

[0083] where:

[0084] P₁ is the probability of receiving a one bit. P₀ is the probability of receiving a zero bit. P_0/1 is the probability of deciding that a received bit is zero when actually it was one. P_1/0 is the probability of deciding that a received bit is one when actually it was zero. To calculate probabilities P_0/1 and P_1/0the probability densities of one and zero signals and decision threshold should be known. $\begin{array}{cc}{P}_{0/1}={}_{-}^D{W}_1\left(x\right)x& \left(4\right)\\ P_{1/0}={}_D^{}{W}_0\left(x\right)x& \left(5\right)\end{array}$

[0085] where:

[0086] D is the decision threshold; W₁ is probability density of one signal; W₀ is probability density of zero signal.

[0087] Hybrid 72 Method:

[0088] For the hybrid x² method the distribution functions and error probabilities are defined as follows.

[0089] The x² probability density function of squared and averaged optical signal and optical noise can be represented by the formula: $\begin{array}{cc}W\left(x\right)=\frac{M}{I_0}{\left(\frac{x}{I_1}\right)}^{\left(M-1\right)/2}\exp \left(-Mx+\frac{I_1}{I_0}\right)I_{M-1}\left(2\frac{\sqrt{x{I}_1}}{I_0}M\right)& \left(6\right)\end{array}$

[0090] where:

[0091] I_m1(.) is a modified Bessel function.

[0092] I₀ and I₁ are two parameters of the x² distribution that between them account for the scaling and non-centrality of the distribution.

[0093] M is an integer, being the number of degrees of freedom.

[0094] The probability density function of optical signal and optical noise with postdetection gaussian noise cannot be expressed in closed form. It can be obtained as an inverse Fourier transform of its characteristic function: $\begin{array}{cc}G\left(\right)=\frac{\exp \left[-\frac{I_1{I}_0}{M-i{I}_0}{}^2\right]}{{\left(1-\frac{i}{M}{I}_0\right)}^M}\exp \left(-\frac{1}{2}{}_p^2{}^2\right)\exp \left(i{I}_0\right)& \left(7\right)\end{array}$

[0095] where: ²_p is the variance of postdetection gaussian noise and i is the imaginary unit.

[0096] Using the steepest descent method the following approximation of probability density function can be obtained: $\begin{array}{cc}W\left(x\right)=\frac{M}{I_0}\frac{\exp \left(F\left(u_0\right)\right)}{\sqrt{2{\left(\frac{{}^{2}F}{u^2}\right)}_{u=u_0}}}& \left(8\right)\end{array}$

[0097] where: $\begin{array}{cc}F\left(u\right)=M\left(\frac{u-1}{u}-\frac{I_1}{I_0}{\left(\frac{u-1}{u}\right)}^2\right)-M\ln \left(u\right)+\frac{1}{2}{\left(\frac{M}{I_0}{}_p\right)}^2{\left(u-1\right)}^2& \left(9\right)\\ {\left(\frac{{}^{2}F}{u^2}\right)}_{u=u_0}=M\left[\frac{2}{u_0^3}\frac{I_1}{I_0}+\frac{1}{u_0^2}+M\frac{{}_p^2}{I_0^2}\right]& \left(10\right)\end{array}$

[0098] and u₀ is maximal real positive root of the equation $\begin{array}{cc}\left(\frac{F}{u}\right)=u^3+u^2\left(\frac{x{I}_0}{M{}_p^2}-1\right)-u\frac{I_0^2}{M{}_p^2}-\frac{I_0{I}_1}{M{}_p^2}=0& \left(11\right)\end{array}$

[0099] For one and zero signals there are two parameter sets of probability density:

[0100] I₀₁, I₁₁, M₁for one signal

[0101] I₀₀, I₁₀, M₀for zero signal.

[0102] Using the symbolism below that index 1 means that in the expression parameter set for one signal is used, and index 0 means that parameter set for zero signal is used, the probabilities (4) and (5) of error can be calculated as

[0103] where $\begin{array}{cc}{P}_k\left(u\right)=\sqrt{\frac{M{A}_k\left(u\right)}{2u^3}}\frac{\exp \left[F_k\left(u\right)\right]}{\left|\frac{u-1}{u^3}M{A}_k\left(u\right)+\frac{B_k\left(u\right)}{u{A}_k\left(u\right)}\right|}& \left(14\right)\\ A\left(u\right)=2\frac{I_1}{I_0}+u+M\frac{{}_p^2}{I_0^2}{u}^3& \left(15\right)\\ B\left(u\right)=3\frac{I_1}{I_0}+u& \left(16\right)\end{array}$

[0104] u_Dk(k0, 1) are the maximum real positive solutions of equation (11) with corresponding indices and xD i.e. $\begin{array}{cc}{\left(\frac{F_k}{u}\right)}_{u=u_{Dk}}=0k=0,1& \left(17\right)\end{array}$

[0105] The optimal decision threshold should be found from the equation

[0106] which can be solved numerically.

[0107] Hybrid Gaussian Method:

[0108] For the hybrid Gaussian method the distribution functions and error probabilities are defined as follows.

[0109] A Gaussian probability density is as follows: $\begin{array}{cc}W\left(x\right)=\frac{1}{\sqrt{2}}\exp \left(-\frac{{\left(x-\right)}^2}{2{}^2}\right)& \left(19\right)\end{array}$

[0110] where:

[0111] ² is total noise variance, which includes optical and post detection noise

[0112] is the mean value of the signal.

[0113] The probabilities (4) and (5) are therefore: $\begin{array}{cc}{P}_{0/1}=\frac{1}{{}_1\sqrt{2}}{}_{-}^D\exp \left(-\frac{{\left(x-{}_1\right)}^2}{2{}_1^2}\right)x=\frac{1}{2}\mathrm{erfc}\left(\frac{{}_1-D}{{}_1\sqrt{2}}\right)& \left(20\right)\\ P_{1/0}=\frac{1}{{}_0\sqrt{2}}{}_D^{}\exp \left(-\frac{{\left(x-{}_0\right)}^2}{2{}_0^2}\right)x=\frac{1}{2}\mathrm{erfc}\left(\frac{D-{}_0}{{}_0\sqrt{2}}\right)& \left(21\right)\end{array}$

[0114] The optimal decision threshold D_opt should be found from equation

[0115] which is calculated as: $\begin{array}{cc}{D}_{\mathrm{opt}}=\frac{1}{{}_0^2-{}_1^2}\left(I_1{}_0^2-I_0{}_1^2-{}_0{}_1\sqrt{{\left(I_0-I_1\right)}^2-2ln\left(\frac{{}_1{P}_0}{{}_0{P}_1}\right)\left({}_0^2-{}_1^2\right)}\right)& \left(23\right)\end{array}$

[0116] Additional Calculations:

[0117] For both the hybrid x² and hybrid Gaussian methods, the following additional quantities may be calculated.

[0118] The effective Q factor should be found from the equation: $\begin{array}{cc}\mathrm{BER}\left(D_{\mathrm{opt}}\right)=\frac{1}{2}\mathrm{erfc}\left(\frac{Q_{\mathrm{eff}}}{\sqrt{2}}\right)& \left(24\right)\end{array}$

[0119] which can be solved numerically.

[0120] The algorithm of Q factor calculation is independent of the assumption made on noise statistics, and can be calculated as: $\begin{array}{cc}Q=\frac{{}_1-{}_0}{{}_1+{}_0}& \left(25\right)\end{array}$

[0121] where, ₁, ₀, ₁, ₀ are mean values and variances (including optical and postdetection noise) of one and zero signals respectively.

[0122] The eye opening (EO) calculation is independent of the assumption made on noise statistics, and can be calculated as:

[0123] Algorithm Implementation

[0124] An overall algorithm 80 can be implemented via the steps 81 to 95 as shown in FIG. 9.

[0125] Initially the input samples representing the optical signal waveform are processed 81 so as to conform to the requirements of the algorithm. The original bit sequence is then processed 82 to correlate the signal samples with the transmitted bits. The sampled data is then arranged 83 into groups of samples corresponding to sampling instants within each bit. Depending upon user requirements, the core calculations 91 to 95 may be carried out for a single set sample time 84, or for all sample times starting with an initial set sample time 84 followed by other sample times according to the loop 85. If a single sample time only is considered, the result corresponding to this sample time is output 87. Otherwise an optimum result can be calculated by selecting the best result 89 from those calculated for each sample time.

[0126] The calculations required to complete these steps are described in greater detail under the following headings:

[0127] 1. Receive the Input Data (81)

[0128] The input electrical sampled signal and the input bit sequence should be received and stored. Denote:

[0129] NSamples is number of electrical samples in the block under processing.

[0130] E_i is an electrical sample, i1 . . . NSamples.

[0131] NBits is number of bits in the bit sequence under processing.

[0132] B_i is symbol of bit sequence i1 . . . NBits. B_i can be either 0 or 1.

[0133] If some electrical samples are negative then an offset x should be added to all samples, according to the following equations:

[0134] 2. Process the Bit Sequence (82)

[0135] Normally it is assumed that received bit sequence consists only of two different symbols 0 and 1, but when ISI is taken into account the preceding and subsequent symbols have impact on the parameters of the signal, representing the current symbol. In this case each symbol of the bit sequence can be considered to correspond to one of eight metasymbols: 000, 001, 010, 011, 100, 101, 110, 111.

Metasymbol	Subsequent Symbol	Current Symbol	Preceding Symbol
0	0	0	0
1	0	0	1
2	0	1	0
3	0	1	1
4	1	0	0
5	1	0	1
6	1	1	0
7	1	1	1
Denote:
MSi is the metasymbol corresponding to i bit in the bit sequence, i 1 . . . NBits.

[0136] The algorithm for transforming a symbol to its corresponding metasymbol is:

[0137] When i1, B_i1B_NBits for periodic boundary conditions and B_i10 for aperiodic boundary conditions.

[0138] When iNbits, B_i1B₁ for periodic boundary conditions and B_i10 for aperiodic boundary conditions.

[0139] Denote:

[0140] NOnes is number of one symbols in the bit sequence.

[0141] NZeros is number of zero symbols in the bit sequence.

[0142] NMS_i is number of i metasymbol in the bit sequence.

[0143] 3. Arrange the Sampled Data (83)

[0144] The sample representing each bit of the sequence can be taken at NSamplesPerBit instants, as shown in FIG. 10 where the sampling instants are drawn as black dots. Consequently, there will be NSamplesPerBit different BER values corresponding to each time instant at which the signals are sampled. Thus it is possible to write BERBER(t_sample), QQ(t_sample), Q_effQ_eff(t_sample), t_sample1 . . . NSamplesPerBit.

[0145] The number of samples per bit is simply related to the sample rate as follows:

[0146] The data samples should be arranged into groups representing each symbol (or metasymbol if ISI is considered), at all possible sampling instants. In other words, for each possible sampling instant we shall have groups of samples corresponding to one and zero bits, or to each of eight metasymbols if ISI is considered.

[0147] 4. Determine the Sample Time (84, 85)

[0148] A single desired sample time for the calculations may be specified by the user. In this case, the sample time is set 84, and the core calculations 91 to 95 are performed for this one sample time. The BER and Q values corresponding to this specific sample time should be the module's output 87.

[0149] Alternatively, and more commonly, an optimum result or results for a range of sample times may be required. In this case an initial sample time is set 84, and then the core calculations 91 to 95 are carried out for all sample times via the loop 85. The BER, Q and Eye Opening values should be estimated at all possible sampling instants according to the decisions 86, 87, and wherein upon each execution of the loop a different sampling time is selected 90. Then the sampling instant at which BER is minimal t_min should be determined and the BER, and Q values corresponding to t_samplet_min should be the module's output 89. If BER takes its minimal value at more than one sampling instant, then from these instants one should be selected at which Q value is maximal. The index of sampling instant relative to the bit frame is determined as

[0150] The sample from which averaging starts (relative to the bit frame) can be determined as:

[0151] The sample from which averaging ends (relative to the bit frame) can be determined as:

[0152] Output values of the module should be calculated as: $\begin{array}{cc}\mathrm{BER}=\frac{\underset{k=t_{\mathrm{start}}}{\stackrel{t_{\mathrm{stop}}}{}}\mathrm{BER}\left(k\right)}{1+t_{\mathrm{stop}}-t_{\mathrm{start}}}& \left(34\right)\\ Q=\frac{\underset{k=t_{\mathrm{start}}}{\stackrel{t_{\mathrm{stop}}}{}}Q\left(k\right)}{1+t_{\mathrm{stop}}-t_{\mathrm{start}}}& \left(35\right)\end{array}$

[0153] 5. Calculate Data Statistics (91)

[0154] For BER, Q, threshold and Eye opening calculations the variance and mean values of data samples corresponding to the one and zero symbols are used. Denote:

[0155] ₁ is the mean value, corresponding to one symbols.

[0156] ₀ is the mean value, corresponding to zero symbols.

[0157] ₁² is the variance, corresponding to one symbols.

[0158] ₀² is the variance, corresponding to zero symbols.

[0159] If ISI is not considered $\begin{array}{cc}{}_1=\frac{\underset{\mathrm{ones}}{}{E}_i}{\mathrm{NOnes}}& \left(36\right)\\ {}_0=\frac{\underset{\mathrm{zeros}}{}{E}_i}{\mathrm{NZeros}}& \left(37\right)\\ {}_1^2=\frac{1}{\mathrm{NOnes}-1}\underset{\mathrm{ones}}{}{\left({}_1-E_i\right)}^2& \left(38\right)\\ {}_0^2=\frac{1}{\mathrm{NZeros}}\underset{\mathrm{zeros}}{}{\left({}_0-E_i\right)}^2& \left(39\right)\end{array}$

[0160] where $\underset{\mathrm{zeros}}{}\mathrm{and}\underset{\mathrm{ones}}{}$

[0161] denote sum over samples representing zero or one symbol taken at the defined time instant t_sample. If ISI is considered then the statistics should be calculated for each metasymbol. And the total statistics for one and zero symbols should be also calculated.

[0162] Denote:

[0163] _i is the mean value, corresponding to i metasymbol.

[0164] _i is the variance, corresponding to i metasymbol. $\begin{array}{cc}{}_i=\frac{\underset{{\mathrm{NMS}}_i}{}{E}_k}{{\mathrm{NMS}}_i}& \left(40\right)\\ {}_i^2=\frac{1}{{\mathrm{NMS}}_i-1}\underset{{\mathrm{NMS}}_i}{}{\left({}_1-E_k\right)}^2& \left(41\right)\end{array}$

[0165] where $\underset{{\mathrm{NMS}}_i}{}$

[0166] denotes sum over samples representing i metasymbol, taken at the defined time instant t_sample.

[0167] 6. Determine the Threshold (92)

[0168] To be able to calculate values of BER the decision threshold D should be known. Depending upon requirements, an absolute threshold can specified by a parameter Threshold.

[0169] Or if relative thresholding is used, the decision threshold can be:

[0170] If a parameter ThresholdType is set to Optimum and ISI is not considered, then for gaussian assumption optimal threshold is obtained as follows:

[0171] If _1<>0 and ₀>0 optimal threshold is calculated in accordance with (23), where $\begin{array}{cc}{P}_1=\frac{\mathrm{NOnes}}{\mathrm{NOnes}+\mathrm{NZeros}}& \left(44\right)\\ P_0=\frac{\mathrm{NZeros}}{\mathrm{NOnes}+\mathrm{NZeros}}& \left(45\right)\end{array}$

[0172] If ₁₀ then optimal threshold is calculated as:

[0173] If ₁0 then

[0174] If ₀0 then

[0175] If the parameter ThresholdType is set to Optimum and ISI is not considered, then for chi-square assumption optimal threshold is obtained as solution of equation (18) with P₁ and P₀ calculated in accordance with (44) and (45). This equation should be solved numerically by the bisection method on the interval D₀, ₁ with tolerance (₁₀)/10000. This procedure requires calculation of the PDF (8). The calculation of the PDF is described below.

[0176] If the parameter ThresholdType is set to Optimum and ISI is considered, then independently on statistical assumption optimal decision threshold is found by numerical minimization of BER_ISI(D) on the interval D₀, ₁ by the method of golden section. Calculation of BER_ISI is described below.

[0177] 7. Calculation of BERisi (93, 94)

[0178] When ISI is considered according to the decision 93 then total BER is calculated 94 as a weighted sum of error probabilities for each metasymbol, according to the following equation: $\begin{array}{cc}\mathrm{BER}\left(D\right)=\underset{I=1}{\overset{8}{}}\frac{{\mathrm{NMS}}_I}{\underset{i=1}{\overset{8}{}}{\mathrm{NMS}}_i}{\mathrm{BER}}_i\left(D\right)& \left(49\right)\end{array}$

[0179] where BER_i(D) is BER for i metasymbol. The BER of each metasymbol is calculated as follows.

[0180] For metasymbols representing logical one (i.e., i2,3,6,7) calculation can be according to (20) if a parameter EstimationMethod is set to Gauss_ISI, and according to (12) if the parameter EstimationMethod is set to Chi2_ISI.

[0181] For metasymbols representing logical zero (i.e., i1,4,5,8) calculation can be according to (21) if parameter EstimationMethod is set to Gauss_ISI, and according to (13) if parameter EstimationMethod is set to Chi2_ISI.

[0182] 8. Calculate the Hybrid x² PDF

[0183] This procedure is required when PDF parameter fitting is performed for the calculation of BER 95 or BERisi 94, and also when optimal threshold is calculated 92. Under this heading, the calculation of the PDF itself is described, assuming its parameters are known. The hybrid x²PDF W(x) has four parameters: I₀, I₁, M, ²_p, the actual calculation of which under various conditions is described under the following heading.

[0184] (a) Calculation of PDF with Negligible Postdetection Noise

[0185] If ²_p0or I₀(2I₁I₀)/M>1000²_p, then equation (6) should be used for PDF calculation. Variance of optical noise is 1000 times larger than variance of postdetection noise, and postdetection noise can be neglected. The method of computing equation (6) can be dependent on the argument and parameter values. If x<0 then

[0186] Otherwise, the first argument of the Bessel function should be calculated: $\begin{array}{cc}\mathrm{argum}=\frac{2M\sqrt{{\mathrm{xI}}_1}}{I_0}& \left(51\right)\end{array}$

[0187] Then if I_m1(argum)0 then logarithm of (6) should be calculated by one of the following:

[0188] If M1>3 argum then $\begin{array}{cc}\ln \left(W\left(x\right)\right)=\ln \left(\frac{M}{I_0}\right)+\frac{M-1}{2}\ln \left(\frac{x}{I_1}\right)-\frac{M\left(x+I_1\right)}{I_0}-\mathrm{gammaln}\left(M\right)+\left(M-1\right)\ln \left(\frac{\mathrm{argum}}{2}\right)& \left(52\right)\end{array}$

[0189] where gammaIn(M) is logarithm of gamma function, the calculation of which is more fully described in William H. Press et al., Numerical Recipes in C: The Art of Scientific Computing, 2^nd ed.

[0190] If argum>3(M1) then $\begin{array}{cc}\ln \left(W\left(x\right)\right)=\ln \left(\frac{M}{I_0}\right)+\frac{M-1}{2}\ln \left(\frac{x}{I_1}\right)-\frac{M\left(x+I_1\right)}{I_0}+\mathrm{argum}-\frac{\ln \left(2\mathrm{argum}\right)}{2}& \left(53\right)\end{array}$

[0191] If argum<3(M1) and M1>3argum then

[0192] If I_M1(argum), i.e., overflow occurs during calculation (this condition can be replaced with argum>700), then a scaled Bessel function can be used.

[0193] and the PDF is calculated as:

[0194] If BS_M1(argum)0 then

[0195] If BS_M1(argum)<>0 then $\begin{array}{cc}\ln \left(W\left(x\right)\right)=\ln \left(\frac{M}{I_0}\right)+\frac{M-1}{2}\ln \left(\frac{x}{I_1}\right)+\frac{M\left(2\sqrt{{\mathrm{xI}}_1}-x-I_1\right)}{I_0}+\ln \left({\mathrm{BS}}_{M-1}\left(\mathrm{argum}\right)\right)& \left(57\right)\end{array}$

[0196] If I_m1(argum)<>0 and I_M1(argum)<>then PDF is calculated in accordance with (6).

[0197] (b) Calculation of PDF with Postdetection Noise

[0198] At first maximal positive real root of equation (11) should be found.

[0199] (1) Solving Equation (11)

[0200] If ² _p0 then (9) becomes quadratic and the root needed is: $\begin{array}{cc}{u}_0=\frac{I_0}{2y}+\sqrt{{\left(\frac{I_0}{2y}\right)}^2+\frac{I_1}{y}}& \left(58\right)\end{array}$

[0201] If ² _p0, then equation (11) is solved by Newton's method as follows.

[0202] The initial value u⁰₀ from which iterations start is found by the following iterative algorithm. First two intermediate values are calculated a, and b. $\begin{array}{cc}a=\frac{{\mathrm{xI}}_0}{M{}_p^2}-1& \left(59\right)\\ b=-\frac{I_0^2}{M{}_p^2}& \left(60\right)\end{array}$

[0203] The starting point for iterations to find u⁰₀ is calculated as: $\begin{array}{cc}\{\begin{array}{c}{u}_0^{0,0}=a\left(-\frac{1}{3}+\sqrt{\frac{1}{9}-\frac{{}_p^{2}M}{x^2}}\right),a>1000\\ u_0^{0,0}=-\frac{a}{3}+\sqrt{\frac{a^2}{9}-b},a1000\end{array}& \left(61\right)\end{array}$

[0204] The iteration algorithm is defined by: $\begin{array}{cc}{u}_0^{0,i+1}=u_0^{0,i}+\left|\frac{F}{u}/\mathrm{div}\left(u_0^{0,i},x,I_{0,}{I}_{1,}M,{}_p^2\right)\right|+& \left(62\right)\end{array}$

[0205] where $=1{10}^{-15},\frac{F}{u}$

[0206] is given in (11) and div(u, x, I₀, I₁, M, ²_p)is calculated as: $\begin{array}{cc}\mathrm{div}\left(u,x,I_0,I_1,M,{}_p^2\right)=3_u^2+2u\left(\frac{{\mathrm{xI}}_0}{M{}_p^2}-1\right)-\frac{I_0^2}{M{}_p^2}& \left(63\right)\end{array}$

[0207] The iterations (62) are performed while $\frac{F}{u}$

[0208] or u<0.

[0209] The iterative procedure for finding u₀ is described by: $\begin{array}{cc}{u}_0^{i+1}=u_0^i-\frac{F}{u}/\mathrm{div}\left(u_0^i,x,I_0,I_1,M,{}_p^2\right)& \left(64\right)\end{array}$

[0210] If div(u¹₀, x, I₀, I₁, M, ²_p)0 then div(u¹₀/2, x, I₀, I₁, M, ²_p)should be used in divisor, where:

[0211] and the initial value of is ₀10.

[0212] The iterations continue while $\left|\frac{}{u_0^{i+1}}\right|>,$

[0213] or until number of iterations exceeds 1000. When u₀ is found, the PDF is calculated using (8), (9), (10).

[0214] 9. Calculation of Hybrid x² PDF Parameters

[0215] In order to compute the x² PDF as described under the preceding heading, the four parameters of the hybrid x² PDF must be calculated from the input data and parameters. The following procedure is used.

[0216] (a) Calculation of the postdetection noise variance

[0217] The postdetection noise variance is given by the equation:

[0218] The shot noise variance ²_s is calculated as:

[0219] where _k is mean value of the data samples representing either one or zero symbol (k1 for the samples representing one symbol and k0 for the samples representing zero symbol) and q is the electron charge. The thermal noise variance ²_th is given by:

[0220] (b) Fitting of I₀, I₁, M Parameters

[0221] To determine parameters I₀, I₁, M, the mean value and standard deviation of data samples can be used. The PDF could be calculated then as function of , , and M, so as to obtain function a depending on one unknown parameter. This parameter should be found by a fitting procedure:

[0222] The maximum possible value of M is $\begin{array}{cc}{M}_{\max }=\mathrm{floor}\left(\frac{{}^2}{{}^2}\right)& \left(69\right)\end{array}$

[0223] A merit function can be constructed so that it is dependent on only one parameter M and then minimized on the interval M1, M_max with tolerance 0.5 (since M is integer). If M_max<1 then a chi-square fit cannot be performed. The merit function is chi-square statistic x². The algorithm of x² calculation is as follows:

[0224] Define:

[0225] n to be the number of samples in the data being fitted.

[0226] Arrange data as a n_b-bins (class interval) histogram: m_i, x_i, i1 . . . n_b, where:

[0227] m_inumber of samples in iclass interval.

[0228] x_icenter of i class interval.

[0229] The number of bins n_b is calculated as: $\begin{array}{cc}{n}_b=\mathrm{floor}\left(\frac{\max \left(\mathrm{data}\right)-\min \left(\mathrm{data}\right)}{0.4\sqrt{{}^2}}\right)& \left(70\right)\end{array}$

[0230] Then the expected frequencies for histogram are calculated.

[0231] where:

[0232] x(max(data)min(data))/n_b is the width of the class interval

[0233] Define the fitting function (x_i) to be the chi square PDF which is calculated depending only upon M using equation (6) with

[0234] If, for some j, np_j<10, then those bins should be combined into one using the following procedure:

[0235] Define:

[0236] R to be the index of the last bin which has expected frequency>10

[0237] L to be the index of the first bin which has expected frequency>10

[0238] Then the right tail T_R is the sum of all bins with indices from R to n_b $\begin{array}{cc}{T}_R=\underset{k=R}{\overset{n_b}{}}{m}_k& \left(74\right)\end{array}$

[0239] The left tail T_L is the sum of all bins with indices from 1 to L inclusive; $\begin{array}{cc}{T}_L=\underset{k=1}{\overset{L}{}}{m}_k& \left(75\right)\end{array}$

[0240] Expected frequencies for tails are calculated as sums of expected frequencies for combined bins. $\begin{array}{cc}\begin{array}{cccc}{E}_R=n\underset{k=R}{\overset{n_b}{}}{P}_k& & & E_L=n\underset{k=1}{\overset{L}{}}{P}_k\\ & & & \end{array}& \left(76\right)\end{array}$

[0241] Then combined bins should not be considered. Note that after combining bins n_b is reduced, so the right tail should be calculated first.

[0242] If number of valid intervals r (number of intervals left after combining bins) is less than 4 then

[0243] The chi-square statistic is calculated as: $\begin{array}{cc}{}^2=\frac{{\left(T_R-E_R\right)}^2}{E_R}+\frac{{\left(T_L-E_L\right)}^2}{E_L}+\underset{i=1}{\overset{r}{}}\frac{{\left(m_i-{\mathrm{np}}_i\right)}^2}{{\mathrm{np}}_i}& \left(78\right)\end{array}$

[0244] After the minimization procedure is finished all parameters of the PDF are found from (72), (73).

[0245] 10. Calculation of the Bit Error Rate

[0246] Given the parameters of the hybrid x² PDF, calculation of the BER 95 or BERisi 94 can be carried out using the equations previously described. If ISI is not considered 95, then BER is calculated using (1) with (44), (45). If ISI is considered 94 then BER is calculated using (49). The conditional probabilities P_0/1 and P_1/0 are calculated according to the following methods for chi-square and gaussian assumptions respectively.

[0247] (a) Hybrid x² Method

[0248] In the chi-square assumption for data representing any symbol (or metasymbol) conditional probabilities as a function of decision threshold D and PDF parameters I₀, I₁, M, ²_p are calculated as:

[0249] If DI₀I₁ then P_k1, k denotes 0/1 or 1/0

[0250] If D>I₀I₁ then P_k is calculated using (12-17).

[0251] The value of P_k should be limited: if it occurs that P_k>1 then P_k1 should be output.

[0252] (b) Hybrid Gaussian Method

[0253] In the prior art method based on the gaussian assumption the conditional probabilities of error are simply calculated according to (20), (21) with:

[0254] where ² is estimated variance of the samples, representing the symbol for which error probability is calculated.

[0255] It will be understood that the invention disclosed and defined herein extends to all alternative combinations of two or more of the individual features mentioned or evident from the text or drawings. All of these different combinations constitute various alternative aspects of the invention.

[0256] The foregoing describes embodiments of the present invention and modifications, obvious to those skilled in the art can be made thereto, without departing from the spirit of the invention or the scope of the appended claims.