Conclusions References

Introduction Conclusions References Tables Figures ◭ ◮ ◭ ◮ Back Close Full Screen / Esc Abstract Introduction Conclusions References Tables Figures ◭ ◮ ◭ ◮ Back Close Full Screen / Esc Abstract The representation of data, whether geophysical observations, numerical model output or laboratory results, by a best fit straight line is a routine practice in the geosciences and other fields. While the literature is full of detailed analyses of procedures for fitting straight lines to values with uncertainties, a surprising number of scientists blindly 5 use the standard least squares method, such as found on calculators and in spread-sheet programs, that assumes no uncertainties in the x values. Here, the available procedures for estimating the best fit straight line to data, including those applicable to situations for uncertainties present in both the x and y variables, are reviewed. Representative methods that are presented in the literature for bivariate weighted fits 10 are compared using several sample data sets, and guidance is presented as to when the somewhat more involved iterative methods are required, or when the standard least-squares procedure would be expected to be satisfactory. A spreadsheet-based template is made available that employs one method for bivariate fitting.


Introduction
Representation of the relationship between x (independent) and y (dependent) variables by a straight line (or other function) is a routine process in scientific and other disciplines.Often the parameters (slope and y-intercept) of such a fitted line can be related to fundamental physical quantities.It is therefore very important that the parameters accurately represent the data collected, and that uncertainties in the parameters are estimated and applied correctly or the results of the fitting process and thus the scientific study could be misinterpreted.
Correspondence to: C. A. Cantrell (cantrell@ucar.edu)The approaches to fitting straight lines to collections of x−y data pairs can be broadly grouped into two categories: the "standard" least-squares methods in which the distances between the fitted line and the data in the y-direction are minimized, and the "bivariate" least-squares methods in which the perpendicular distances between the fitted line and the data are minimized.A third method, similar to the second but less commonly employed, involves minimization of the areas of the right triangles formed by the data point and the line.In all of these methods, weights may be also applied to the data to account for the differing uncertainties in the individual points.In "standard" least-squares, the weighting pertains to the y-variables only, whereas in "bivariate" methods, weights can be assigned for the x-and y-variables independently.There is widely varying terminology for these procedures in the literature that can be confusing to the non-expert.Authors have used terms such as major axis regression, reduced major axis regression, ordinary least-squares, maximum likelihood, errors in variables, rigorous least-squares, orthogonal regression and total least-squares.Herein, the terms "standard" and "bivariate" will be used to denote these two categories of fitting methods.This paper does, however, present a detailed reference list of available methods and applications presented in the literature.
For demonstration and testing purposes, two data sets from the literature were employed.First, the well-known data of Pearson (1901) with weights suggested by York (1966) were used (see Table 1 and Fig. 1).The data values are similar to those that might be encountered in a laboratory study or acquired in atmospheric measurements, but with rather extreme weights that range 3 orders of magnitude as the data ranges about a factor of five.This data set has the advantage that the exact results of the bivariate fit are known and reported in the literature, and one that is frequently used as a test for new fitting methods.
Published by Copernicus Publications on behalf of the European Geosciences Union.Fig. 1.Linear fits to the data of Pearson [1901] with weights suggested by York [1966] ("Pearson-York" data set, shown in Table 1).
The weights have been plotted as σ values (w i =1/σ 2 i ).Fit parameters are shown in Table 2. Table 1.Example data "Pearson's data with York's weights" for comparison of fitting procedures described in the text.A second data set was created by selecting random numbers from Gaussian distributions and adding them to base values, which were numbers 1 through 100 (see Fig. 2).Initially, the Gaussian distributions were set with means of zero, and standard deviations of 10 units plus 30% of the base value, but other tests were performed with different amounts of constant and proportional uncertainty.These data were meant to represent those that would result from an intercomparison of two instruments measuring the same quantity, which have baseline noise of 10 units (1 sigma), measurement uncertainties that are well above the baseline of 30% (1 sigma), and nominal "true" values from 1 to 100 units.This data set has the characteristic that in the absence of noise, or if the noise is properly dealt with, the best fit line should have a slope of one, and an intercept of zero.Fig. 2. Linear fits to data generated by sampling a Gaussian function with standard deviation of 10 units plus 30%, and adding the noise to the numbers 1 through 100.Fit parameters are shown in Table 3.
Next, the methods were applied to two examples of authentic data to demonstrate specifically the value of bivariate methods, and to point out how and when they should be applied.
This review and recommendation does not attempt to be mathematically nor statistically rigorous.The reader is referred to the referenced literature for such details.The purpose here is to provide operational information for the scientific user of these routines, and to provide guidance for the choice of routine to be utilized.
Note that there is not universal agreement in the uses of symbols for the measured x and y values and the calculated slope and intercept that appear in the literature.The reader is cautioned in this regard.In this paper, x i and y i (lower case italics) refer to the measured x and y values, m refers to the slope of the best fit line, and b is the y-axis intercept.Other symbols are defined throughout the paper.

Standard least-squares
The equations for a line that best describes x−y data pairs when all of the measurement error may be assumed to reside in the y-variable (i.e. the x values are exact or nearly so) is readily available and easily derived (e.g.Bevington, 1969).The fitted line then becomes a "predicted" value for y given a value for x.The usual method involves minimizing the sum of squares of the differences between the fitted line and the data points in the y-direction (although minimization of other quantities has been used).The slope, m, and y-intercept, b, of this best-fit line can be represented in terms of summations of computations performed on the n measured The symbols refer to the summation of the quantity over all n values, and the subscript, i, denotes the individual measured x and y values.The uncertainties in the slope and intercept can also be calculated.
Another useful quantity is the correlation coefficient (also called the Pearson Correlation Coefficient), which provides an index of the degree of correlation between the x and y data.
It is usually the case that not all the data points have the same uncertainty.Thus, it is desired that data with least uncertainty have the greatest influence on the slope and intercept of the fitted line.This is accomplished by weighting each of the points with a factor, w i , which is often assumed (and demonstrated mathematically to yield the best unbiased linear fit parameters, if set) equal to the inverse of the variance of the y-values (σ 2 yi ).It could include estimates of all sources of uncertainty in the y-values.Other weighting procedures are also possible.The formulas for the slope and intercept are modified as shown to include data weights.
These formulas are readily programmed, or exist as available spreadsheet or calculator functions, and can be routinely applied to fitting of straight lines to x−y data sets.The standard least-squares method was applied with and without weights, using Eqs.(1) and (4), to the two test data sets ("Pearson-York" and "synthetic data") for comparison with the bivariate methods (see Tables 2 and 3).Note that when there are significant x and y errors, that standard leastsquares yields erroneous slopes.For the "synthetic data", the slope was usually too small, whereas for the "Pearson-York" data, the slope was too large (compared to the Williamson-York and Neri et al. methods, discussed below).
In assessing the impacts of errors on linear fits, normal (Gaussian) distributions of the errors are assumed.This is a reasonable assumption for most real-world situations, but it should be recognized that formulations for error estimates of the slope and intercept of the fits will be different for other error distributions.
Some representative examples of exact and approximate procedures (discussed below) from the literature were applied to the sample data sets, and the results of the fits are shown in Tables 2 and 3.In each case, slopes and intercepts were derived by fitting y on x, and by exchanging the x and y variables, thus fitting x on y.The slopes and intercepts for the latter case were made comparable to those of the former case by calculating the equivalent values for y=mx+b (since x=y/m−b/m, then m =1/m and b = − b/m).For methods that properly account for errors in both variables, the fit parameters by these two approaches should be identical (i.e.m from fitting x on y should equal m from fitting y on x, and similarly for b and b).This is termed invariance to exchange of x and y.Proper fitting methods should also be invariant to change of scale (i.e.fit parameters do not depend on the choice of units for x and y).Several numerical digits are shown in Tables 2 and 3, not all significant, so that the results from the various methods can be accurately compared.
The method described by York (1966;1968) and York et al. (2004) was applied to the sample data sets.This involves iteratively solving the following equations (Eq.5).This method allows for correlation between the x and y errors, indicated by r i (different than the r xy in Eq. 3), which is set to zero in the present case (i.e.errors are assumed to be uncorrelated).
The procedure is to assume a starting value for m, calculate W i , U i , V i , α i , and β i , and then calculate a revised value for m.This process is repeated until m changes by some small increment according to the accuracy desired.This is a simpler implementation of an earlier method of York (1966), which was described in York (1969) and York et al. (2004), and is the same as the method of Williamson (1968), if the x and y errors are uncorrelated (i.e.r i =0).The method of Williamson (1968) has been praised in the literature (Mac-Taggert and Farwell, 1992;Kalantar, 1990) as being efficiently able to converge to the correct answer.Other approaches (including the earlier York method), may not always converge or may be slow to do so, depending on the specific data set.As with standard least-squares, one can perform bivariate fits without weighting.This is done by making all the weights the same (e.g. 1).
The quantity S (n − 2) is a "goodness of fit" parameter.Its expected value is unity.Its deviation from unity can be used to adjust the weighting factors (in a global sense), but the bivariate slope and intercept will not be affected.Another straightforward method is that of Neri et al. (1989).This involves minimization of the shortest distance between the fitted line and that data points, and assumes the x and y errors are uncorrelated.The following equations are utilized.
In this method, an initial m is guessed (such as from standard least-squares or by inspection), b is calculated (second equation in Eq. 7), and then m is adjusted to minimize the left hand side of the first equation in Eq. ( 7).The process is repeated until the left side of the first equation in Eq. ( 7) is satisfactorily close to zero.The Williamson-York and Neri et al. methods give identical results for the slope and intercept of the two test data sets.
Four other methods give results that are reasonably close to the above results, but are not exactly the same, and do not always give the same slope on exchange of the x and y variables.These approximate methods may be satisfactory for many applications.Reed (1992) suggests finding roots of the following quadratic expression.
This equation is solved for m by the quadratic formula, , where the choice of roots is refined by comparison with standard least-squares or by inspection.Macdonald and Thompson (1992) describe a number of cases for which their method is applicable.They have made available a FORTRAN program that applies their procedures, which provides nearly exact results for the Pearson-York data set.Similarly, Lybanon (1984) presents a detailed method that also yields results very close to those of the "exact" methods.Krane and Schecter (1982) put forward a method proposed by Barker and Diana (1974) and discussed by others (Irwin and Quickenden, 1983;Orear, 1984;Lybanon, 1984b) that is called "effective variance".One begins with Eq. ( 4), but the weights, w i , are adjusted to the following form.
This is same as York's W i value with uncorrelated errors.Since m appears in the formula for the weight, an iterative process is required, in which an initial m value is guessed, w i is calculated, followed by calculation of a revised m.The result differs from the "exact" methods for the Pearson-York data set by a few percent, but it is more accurate than the standard least-squares.This method does not retrieve the same slope and intercept when the x-and y-variables are exchanged.The errors are larger with the "synthetic" data set.Press et al. (1992a,b) present a method called "maximum likelihood estimation" and include a routine written in C or FORTRAN for its implementation.This is also discussed by Titterington and Halliday (1979).York et al. (2004) demonstrate that their method and "maximum likelihood estimation" are mathematically identical.Brauers and Finlayson-Pitts (1997) applied the Press et al. method to analysis of kinetic data.
The methods of Williamson (1968), York (1969), York et al. (2004) and Neri et al. (1989) all agree and appear to provide the exact answer to the best fit for the Pearson-York data set.The approaches of Reed (1992), Macdonald and Thompson (1992), and Lybanon (1984) provide results very close to the exact ones.The "effective variance" method performs reasonably well for the Pearson-York data set, but poorer for the synthetic data.Because of this variability in performance, it should be used with caution.

Comparing the methods
A more detailed examination of the behavior of bivariate and standard least-squares as a function of the random noise added in the "synthetic data" was performed.The purpose here is to advise the reader when the more involved bivariate methods should be used or when the standard least-squares are expected to provide satisfactory values for the fit parameters.A series of calculations was performed in which random noise was sampled from Gaussian distributions with varying constant and proportional standard deviations (like the second test data set used above   3).Errors in both the x and y variables lead to systematic errors in the slope from standard least-squares.Slopes from the bivariate method show no such systematic variation with r.
weights) were applied to the data sets, as was the method of Williamson-York.The values of r (Eq. 3) were also calculated.This test has the advantage that the "correct" slope and intercept are known (1 and 0, respectively).Note that the errors are normally distributed, which may not necessarily be the case in "real" data sets.
Figure 3 shows m fit /m expected versus r of the best fit lines using standard least-squares when proportional uncertainties of zero to 50% and/or constant uncertainties of up to 50 units were applied to the x-data, the y-data, or both.Standard least-squares performs well by retrieving slopes close to unity (the expected value) when errors are applied to the ydata only.However, when errors are added to the x-variable either alone or with errors added to the y-variable, the slopes of the best fit lines from standard least-squares are significantly less than unity.The ratio of the fitted slope to that expected is approximately equal to |r|.This is true even when r values are very small.The normal interpretation of these small r values is that the data are uncorrelated and cannot be represented by a linear relationship.In this case, however, we know that there is a linear relationship between x and y because of the way the data were constructed.The F statistics for the standard fits indicate that they are statistically significant at the 95% confidence level for all but those corresponding to the 3 smallest r values.
Applying the Williamson-York bivariate method to the same data sets, leads to slopes within about 20% of the expected value of unity.Note that this is the case even when the data are very noisy and thus correlation coefficients are small.Values much closer to the expected value are retrieved when the data is less noisy (see inset in Fig. 3).These fits were performed with 100 data points.If the sizes of the data sets are increased, the error (scatter) in the slope decreases accordingly.As an example, for a constant error of 28 units, the average error in the slope (5 repetitions) decreases from 19% to 6% to less than 1% as the number of data points goes from 100 to 1000 to 10 000 (an approximate √ n relationship).
Knowing that the bivariate methods are an improvement over standard least-squares when there are errors in the xvariable is a start, but can the information gathered be used to indicate when the extra trouble of the bivariate fit is called for, versus when standard least-squares will suffice.Figure 3 shows that there is a rather robust relationship between the systematic error in the slope from standard least-squares and the absolute value of the correlation coefficient (as expected, comparing Eqs. 1 and 3).For errors in both variables, the fractional error in the standard least-squares slope is approximately 1-|r|.Thus, a quick calculation of the correlation coefficient can give a rough indication of the error in the derived standard least-squares slope for data with comparable errors in both variables.If this error in the slope is outside the needs of the task at hand, then a bivariate approach should be employed.For unusual weighting situations (such as the Pearson-York data), it is probably best to always use robust bivariate methods, since the impact of such weights on the fit parameters is not intuitive (although in this specific case, the standard least-squares slope is only in error by 12%).When the error in the y-variable is much greater than the error in x-variable, then standard least-squares performs better than indicated by the calculated r value.

Application to actual observations
Two authentic sets of data from the TRACE-P campaign (TRansport And Chemistry Experiment -Pacific) were selected for application of these fitting procedures.TRACE-P involved two aircraft (the NASA DC-8 and P3-B) as platforms for observations primarily in the western Pacific Ocean basic.The observations used here are gas-phase formaldehyde (CH 2 O) concentrations collected by Alan Fried and colleagues aboard the NASA DC-8 aircraft (Fried et al., 2003), and peroxy radical concentrations (HO 2 +RO 2 ) collected by the author and colleagues aboard the NASA P-3B aircraft (Cantrell et al., 2003).These data represent very typical situations that might require the fitting procedures discussed here.
The details of the measurement techniques and the modeling approaches can be found in the references cited above.Briefly, CH 2 O was measured in the NASA DC-8 aircraft in a low-pressure cell with multi-pass optics (100 m path total optical path) using a tunable lead salt diode infrared laser as the source.A spectral line near 2831.6 cm −1 was scanned and the second harmonic spectrum (after subtraction of the background) was related to the ambient concentration Atmos.Chem.Phys., 8, 5477-5487, 2008 www.atmos-chem-phys.net/8/5477/2008/through addition of known mixtures of CH 2 O in zero air to the instrument inlet.The measurements were corrected for a small interference from methanol.The estimated uncertainty of the measurements was 15%, and detection limits typically ranged from 50 to 80 pptv (parts per trillion by volume).One minute average retrieved concentrations ranged from -47 to 10 665 pptv.Concentrations measured below the detection limit were used as observed in the fits described here.HO 2 +RO 2 concentrations were measured on the NASA P-3B aircraft and were determined by conversion to gas-phase sulfuric acid through the addition of reagent gases NO and SO 2 to the instrument inlet.The sulfuric acid product was ionized by reaction with negatively charged nitrate ions.The product and reagent ions were quantified by quadrupole mass spectrometry.Calibrations were performed using quantitative photolysis of water vapor at 184.9 nm.The estimated uncertainty for these data was 17% and the detection limits were 2-5 pptv.Concentrations below the detection limit were used as observed in the fits described here.
CH 2 O and HO 2 +RO 2 concentrations were estimated by a photochemical box model with inputs of key parameters constrained by the observations (Crawford et al., 1999;Olson et al., 2004).The time-dependent model is run for several days to diurnal steady state.Monte Carlo calculations yielded uncertainty estimates of 20% for modeled CH 2 O and 30% for HO 2 +RO 2 .
Figure 4 shows the measured CH 2 O concentrations versus those estimated by the constrained box model on linear scales (4466 data pairs).The inset plots show the high range of concentrations (>500 pptv, lower right) and the data plotted on logarithmic scales (upper left).The lines represent different methods of fitting the data.The solid line is a weighted bivariate fit to all of the data with the measurements weighted using a variance of the square of 15% of the concentration plus 50 pptv, and the model results using a variance of the square of 20% of the concentration.The slope is near unity (1.054, standard error=0.0144)and the y-intercept is small (1.283, standard error=2.046), in agreement with assessments by Fried et al. (2003) and Olson et al. (2004).The correlation coefficient squared, r 2 is 0.856.The long dashed line is a standard unweighted least-squares fit which yields a slope of 1.462 (standard error=0.0090)and a y-intercept of -44.6 (standard error 3.16).It appears that the line is being unduly weighted by the handful of points at high concentrations in which the model systematically underestimates the observations, leading to a larger slope than the bivariate method.The medium dashed line is a weighted least-squares fit (Eq.4), with weights calculated using the "effective variance" method (Eq.9).Its slope is 0.873 and the y-intercept is 20.1.Finally, the short dashed line is a weighted least-squares fit (equation 4) with weights in the ydirection only (i.e.w i =w yi ).The slope for this fit is 0.811 (std err=0.012)and the y-intercept is 22.4 (std err=2.14).These fits mostly have slopes of unity within the combined measurement-model uncertainties (0.25, 1σ ), with the ex-   P campaign (after Fried et al., 2003;Olson et al., 2004).The data points are divided into two groups: those corresponding to measurements below 500 pptv (small points), and those for measurements above 500 pptv (large points).The main window (on linear scales) shows results of linear fits using four approaches: solid line, bivariate weighted fit to all data; long dash, standard unweighted leastsquares fit; medium dash, fit using weighted standard least-squares (Eq.4) with weights calculated using effective variance; and short dash, fit using weighted standard least-squares with weights in the y-direction only.The lower right inset shows the fit lines and data on expanded x-and y-scales (linear).The upper left inset shows the full range of data on logarithmic scales.See text for fit parameters and discussion.
ception of the standard unweighted least-squares fit.The intercepts are all within the detection limit of the measurements (around 50 pptv).The large slope retrieved with the standard unweighted approach could lead one to make the assessment that there are missing processes in the model, errors in the measurements, or both.While it does appear that there are statistically significant differences between the measurements and the model at high concentrations, the small number of outliers should not significantly change the fit of the entire data set.Eliminating data pairs with measurements greater than 4000 pptv, results in bivariate fit slope and y-intercept values of 1.041 and 2.476, respectively.The weighted standard fits change by small amounts as well.The unweighted standard fit, though, yields slope and y-intercept values of 1.248 and -5.744, respectively.This is a significant change and shows how susceptible the standard fit is to a small number of outliers (the term outlier is used here to mean data that are not described well by the bivariate fit line).
The impact of outliers on the various fit methods is demonstrated further.To the full data set are added numbers of data pairs (up to 1000) for which x is 50 and y is 5000.A second trial added data pairs with x values of 5000 x and y values  of 50.These results are summarized in Fig. 5.It can be seen that outliers above the fit line have little impact on the bivariate and the other weighted fit slopes, even when the number of outliers approaches 20% of the data.The standard unweighted least-squares fit is affected moderately by outliers above the fit line.Outliers below the fit line impact all of the fits greatly except the bivariate.In fact, as shown before, the bivariate fit procedure continues to perform well even when the r 2 parameter indicates that the x and y data are completely uncorrelated.While there have been vari- Fig. 6.Fits of HO 2 +RO 2 measurements versus constrained box model estimates.The lines are four different fit approaches: solid line, bivariate weighted fit to all data; long dash, standard unweighted least-squares fit; medium dash, fit using weighted standard least-squares (Eq.4) with weights calculated using effective variance; and short dash, fit using weighted standard least-squares with weights in the y-direction only.
ous techniques put forward to eliminate outliers (e.g. the Qtest, Dean and Dixon, 1951) that can applied, these exercises show that the bivariate fit method is relatively insensitive to outliers.
As mentioned earlier, and discussed by Fried et al. ( 2003), there appears to be a change in the ratio of measurement to model values from near unity at lower concentrations to well above unity at higher concentrations.As one approach, the data were separated into two groups for measured values below and above 500 pptv, and each group was fit separately.The bivariate slope of the low concentration group is 0.789, while the bivariate slope of the high concentration group is 1.403.An alternate method is to fit the ratio of measurement to model ([CH 2 O] meas /[CH 2 O] model ) versus measurement value.Separating into two groups as before leads to a bivariate slope of 0.00607 for the low concentration group (i.e.moderate dependence of the ratio on the concentration) and an intercept of 0.797 (the ratio at the limit of zero concentration).The slope for the high concentration group is 0.000679 and the intercept is 1.290.It seems that there could be atmospheric processes missing from the model or instrumental issues affecting the measurements in the high concentration regime that need to be addressed (in agreement with Fried et al., 2003).
Fits of measured versus modeled HO 2 +RO 2 are shown along with the data in Fig. 6.The solid line is a bivariate fit weighted using variances for the measurements that are the square of 20% of the concentration plus 5, and using variances for the model results that are the square of 30% of model values.Its slope is 0.961 (std err=0.015)and the intercept is -2.96 (std err=0.35).The correlation coefficient squared, r 2 , is 0.437.The other methods (effective variance, y-weighting only, and no weighting) yield smaller slopes (0.63 to .71).There are some noticeable outliers in which the measured concentrations are systematically higher than the modeled ones at low modeled concentrations.Elimination of these data does not greatly affect the bivariate fit.A fit of the measured to model ratios versus measured values yields a moderate slope (-0.00864) and an intercept near unity (1.053).
It has been reported (Faloona et al., 2000) that measured peroxy radical concentrations are systematically greater than model values at high NO x concentrations.This is observed for TRACE-P HO 2 +RO 2 data as well.For NO concentrations less than 500 pptv, the measured to modeled ratios are close to unity with no significant dependence on NO concentration.The bivariate fit yields a slope of -0.00145 and a y-intercept of 0.77.For NO concentrations greater than 500 pptv, there is a systematic dependence of the measuredmodeled ratio on the NO concentration.The bivariate fit slope is 0.00317 and the y-intercept is -0.785.It has been suggested (Olson et al., 2006) that this phenomenon could be the result of short term large spikes in the NO concentration that impact the average NO concentration, but have little impact on the average peroxy radical concentration.Without high rate NO and peroxy radical data, we cannot rule out such an explanation.Alternatively, there could be unknown photochemical processes or instrumental issues that occur in the presence of high NO concentrations.The measurement and modeling communities continue to search for satisfactory explanations of these observations under high NO conditions.Does the quality of fits obtained with the bivariate methods depend strongly on the selection of weights?This was examined using the CH 2 O measurements and model results.The best estimate for the variance of the measurements is (0.15×[CH 2 O] meas +50) 2 , and for the model values is (0.20×[CH 2 O] model ) 2 .Varying the measured variance values from (0.10×[CH 2 O] meas +50) 2 to (0.30×[CH 2 O] meas +200) 2 results in bivariate fitted slopes ranging from 0.88 to 1.15.Thus, while there is some impact on the fit parameters by the choice of weights, the dependence is not strong.Obviously, every effort should be made to correctly estimate the weights, but small errors in these parameters are not likely to invalidate the fit results.

Summary
Scientists need to use care in applying fitting programs to derive parameters that summarize their observations.In the case of linear fits, significant errors in slopes and intercepts can result using standard least-squares methods if there are uncertainties in the x-values (as cautioned many times in the literature).If the x-and y-variable errors are comparable, 1-|r| may give an indication of the fractional error of the derived standard least-squares slope.If a more accurate slope is desired, then bivariate methods such as those reported by Williamson et al., York et al., or Neri et al. are recommended.For these methods, the accuracy of the slope improves with the number of data points (not so with the standard leastsquares with significant errors in the x-variable).

Fig. 3 .
Fig. 3. Ratio of fitted to expected slopes (m fit /m expected ) from standard least-squares and the Williamson-York bivariate method versus r-values from Eq. (3).Errors in both the x and y variables lead to systematic errors in the slope from standard least-squares.Slopes from the bivariate method show no such systematic variation with r.

Fig. 4 .
Fig. 4. Comparison of measured formaldehyde concentrations with those estimated from a constrained box model during the TRACE-P campaign (after Fried et al., 2003;Olson et al., 2004).The data points are divided into two groups: those corresponding to measurements below 500 pptv (small points), and those for measurements above 500 pptv (large points).The main window (on linear scales) shows results of linear fits using four approaches: solid line, bivariate weighted fit to all data; long dash, standard unweighted leastsquares fit; medium dash, fit using weighted standard least-squares (Eq.4) with weights calculated using effective variance; and short dash, fit using weighted standard least-squares with weights in the y-direction only.The lower right inset shows the fit lines and data on expanded x-and y-scales (linear).The upper left inset shows the full range of data on logarithmic scales.See text for fit parameters and discussion.

Fig. 5 .
Fig.5.Impacts of added data outliers to the formaldehyde dataset presented in Fig.4.Shown are slopes (top panel), intercepts (middle panel), and correlation coefficients (bottom panel) of various fits as impacted by adding extra points, in amounts indicated on the x-axis, to the dataset that are clearly outliers.Eight collections of fit parameters are shown for 1, 10, 100, and 1000 outliers added.Four collections had outliers equal to x=50, y=5000 (dark gray); the other four had outliers equal to x=5000, y=50 (light gray).The circles in the top two panels represent parameters derived from weighted bivariate fits; the downward pointing triangles represent parameters derived from Eq. (4) using effective variance; the squares represent parameters derived from Eq. (4) with weights in the y-direction only; and the diamonds represent parameters derived from unweighted standard least-squares.The values on the y-axis (corresponding to x=0.8) are those derived from the original formaldehyde data with no added outliers.

Table 2 .
Comparison of fit parameters using various weighting and fitting procedures for Pearson's data with York's weights (reproduced in Table1).

Table 3 .
Comparison of fit parameters using various weighting and fitting procedures for synthetic data with random errors (see text).