How curve.fit solves and checks a fit

The measurement uncertainties choose the statistical problem. The solver then estimates the model parameters, checks the result and, when necessary, attempts a bounded recovery. Recovery can change fitted parameters and their uncertainties as well as residuals and statistics. It is more than a display correction.

This page describes the fitting policy introduced through version 2.19.0. Try the synthetic examples to inspect its behavior. For controls and data entry, see application Help.

1. Choose the statistical problem

These are uncertainty modes, not different models. A linear, Gaussian or custom equation uses the objective selected by its uncertainty settings.

Flow chart: from measurements to an objective

Validate the data, model, guesses, fixed parameters and limits.

Which measurement uncertainties are supplied?

Neither x nor yUnweighted least squares.
Keep x fixed and minimize squared vertical discrepancies.
Y onlyWeighted least squares.
Keep x fixed and divide vertical discrepancies by their y uncertainties.
X onlyHorizontal-error fit, with y treated as exact.
A small, positive synthetic y uncertainty represents this constraint numerically. It is not a measurement.
X and yOrthogonal distance regression (ODR).
Allow corrections in both coordinates, weighted by the supplied x and y uncertainties.

Solve the selected problem, then apply its acceptance and recovery checks.

For x+y fits, the objective is the sum of squared standardized corrections:

Here β contains the model parameters, δ is the horizontal correction, and σx and σy are the supplied standard uncertainties. ODR measures distance in uncertainty units, not in screen pixels. A point can have a large horizontal correction and a very small vertical correction.

Residual plots follow the uncertainty mode: no-error and y-only fits show vertical residuals in y units; x-only fits show horizontal residuals in x units; x+y fits show signed effective distances in standard-uncertainty units. In the x+y case, the sum of squared effective residuals should equal Q. If a vertical correction rounds to zero while the horizontal distance is nonzero, the local curve slope determines the residual direction. This preserves the full distance without another solve. A direction that cannot be established produces a precision failure.

2. Assess the result and recover when needed

The chart summarizes decision policy. Checks are also repeated during parameter-limit refinement and confirmation solves. Invalid trial points can require a fresh confirmation. Every path remains subject to cancellation and a finite work and time budget.

Flow chart: acceptance checks and targeted recovery

Initial solve, honoring fixed parameters and limits.

What do the solver status and numerical checks establish?

Normal acceptance policy is satisfied
Keep the native solution and its applicable warnings.
For ordinary measured-error and unweighted fits, an inconclusive check alone does not replace a converged result. It does not prove optimality.
Return the result with its status and warnings.
Convergence recovery is required
An iteration limit, a demonstrated inconsistent solution, or failed x-only acceptance checks triggers the appropriate method:
- X only: progressively tighten a synthetic y uncertainty to the original final anchor.
- X+y: temporarily enlarge y uncertainties, then return to the supplied values.
- Y only / neither: restart from the returned parameters at unchanged weights.
↓
Verify the final solution. For x+y, also repeat the precision screen at the original uncertainties; unresolved corrections lead to the inverse recovery below.
X+y corrections lose meaningful numerical precision
Use local inverse-coordinate recovery.
Swap coordinates and uncertainties together. Solve the same weighted objective with the original parameter definitions.
Verify the inverse solution, polish point projections, confirm covariance where parameters are estimated, and reconstruct consistent corrections in the original coordinates.
Recovery verification or a hard requirement fails
An unusable model, an unverified recovery, a deadline or an exhausted work allowance stops the fit with guidance.
A finite parameter value alone is not sufficient evidence for success.
Report the failure; keep the submitted inputs available for correction.
Recovery is accepted only after its required checks pass. Check constraints, finite and consistent covariance, local optimality conditions, numerical precision and applicable parameter/covariance confirmation. Existing explicitly rank-deficient results retain their warning semantics; they are not certified full-rank recoveries.

Why continuation and inverse recovery are different

Continuation makes an initially difficult optimization easier. It progressively restores the original weights, so it cannot leave enlarged y uncertainties in an accepted x+y result. It can recover a better solution, but it cannot restore digits lost when two almost equal floating-point values are subtracted.

Inverse recovery addresses that arithmetic limit. It is triggered when a meaningful horizontal correction accompanies a vertical correction at the model's evaluation resolution, and that ambiguity matters relative to the supplied y uncertainty. Small numerical values alone do not trigger it; neither does an ordinary poor fit. It may run even when the original solver reports convergence.

On a regular local branch, write x = g(β, y). Swapping the coordinates and their uncertainties preserves the objective:

This is still measured-error ODR. It neither discards y uncertainty nor substitutes an unweighted regression. The implementation resolves the larger standardized correction and reconstructs the smaller one from the pointwise optimality relation, checking consistency with the original curve and objective. Parameters and uncertainties can change when the original answer was inaccurate.

Boundaries of the method

The inverse must be locally usable around every observation's projection. Turning points, ambiguous branches, severe expression cancellation and rank loss can still prevent a verified answer.
A local consistency check is not proof of the global minimum. No multistart recovery or global-minimum search is performed.
Inconclusive diagnostics preserve ordinary converged results and their warnings. They are not a claim that every accepted result was independently certified.
Every method shares the request's existing deadline. A difficult original solve can leave insufficient time for recovery. Additional iterations cannot create missing floating-point precision.
X-only recovery approximates exact y at a fixed numerical anchor. It does not currently invoke the measured x+y inverse recovery. Y-only fits keep x fixed.
Changing units consistently should preserve the statistical problem; extreme offsets, cancellation and finite precision still impose limits. These examples test specific cases, not every equation or possible scale.

3. Try the examples

All seven datasets below are invented numerical demonstrations, not student observations, production data or certified experimental measurements. Their deliberately extreme uncertainties test arithmetic; they are not recommendations for assigning measurement errors.

Each Load example link opens a fresh editable session in a new tab. Your current workspace stays in its original tab. Loading does not run a fit: press Fit!, then inspect the fitted parameters, residuals and χ² in the metric selector or Browser Report. These examples use Absolute parameter uncertainties. Model guesses and fixed settings are loaded with the data.

Accuracy and elapsed time are different kinds of performance. The targets below describe accuracy. Compare responsiveness by fitting the same example more than once; first use, network time and report generation can affect what you observe. No fixed completion time or automatic recovery-path indicator is promised.

A. Difficult x-only line

Three points: x = [0, 1, 2], y = [1, 4, 2], with σx = 0.1 and no y uncertainty. These conflicting points make horizontal regression substantially different from vertical regression.

Inspect: slope ≈ 4.6666667, intercept ≈ −2.3333333 and χ² ≈ 178.571429. The independent reference is weighted linear regression of x on exact y, transformed back to y = mx + b. This small dataset has only one residual degree of freedom. X-only residuals are displayed in x units: divide each by its x uncertainty before squaring and summing to compare with χ².

Load x-only example

B. Measured x+y continuation

The same three points, now with σy = 10⁻⁸. Both uncertainties remain part of the final objective. Equal-uncertainty Deming regression provides an independent reference.

Inspect: slope ≈ 4.6666667, intercept ≈ −2.3333333 and χ² ≈ 178.571429. The values round like the x-only limit because y uncertainty is extremely small, not because it was discarded. Squaring and summing the effective residuals should recover χ²; this is an additional display-consistency check. The solver may take a different recovery path depending on derivative mode.

Load x+y example

C. Precision-limited residuals and a unit-change pair

Eight invented points use x = 1…8, y = 2 × 10⁻¹¹ x + 3 × 10⁻¹¹ plus a specified small perturbation, σx = 0.1 and σy = 10⁻²⁷. The second example multiplies every y value and y uncertainty by 10¹¹.

Inspect: both fits should have the same χ² and standardized residuals. Slopes, intercepts and their standard uncertainties in the rescaled example should be 10¹¹ times those in the original example. Residuals should contain both signs. This checks inverse recovery and unit consistency, including statistics that may look wrong even when the plotted line looks plausible.

Load tiny-y example Load rescaled example

D. Mixed y-uncertainty scales

Eight line observations alternate σy = 0.1 and 10⁻¹⁶, all with σx = 0.1. This exercises per-point precision handling without replacing the different measurement weights.

Inspect: slope ≈ 1.9871429524, intercept ≈ 3.0389107221 and χ² ≈ 6.5258187943. The reference independently profiles the straight-line distance objective. A single error floor applied to all points would change this problem.

Load mixed-uncertainty example

E. A nonlinear curve with two local inverse branches

Twelve quadratic observations span both negative and positive x. The linear coefficient is fixed at zero, with starting quadratic and constant coefficients 2 and 3. No observation is at the turning point.

Inspect: both sides remain on their local projection branches, the linear coefficient stays fixed, and χ² ≈ 0.0432524383. The independent check computes the two signed square-root branches. This demonstrates a nonlinear case that works; it does not establish that inversion is safe at the parabola's turning point.

Load two-branch example

F. A deliberate numerical limitation

Eight line observations have x near 10⁸, y near 5…19, and σy = 10⁻¹⁶. The model evaluates a large slope-times-x term plus a nearly cancelling intercept. These supplied uncertainties demand more reliable digits than this expression provides.

Inspect: the current method reports a precision failure rather than presenting a rounding artifact as reliable χ². Failure is the expected demonstration here. Review the origin and precision of your measurements; changing weights merely to obtain a success message is not a justified remedy.

Load cancellation example

For a published experimental comparison with ordinary x+y uncertainties, use the Pearson–York example. For the distinction between reference datasets and synthetic checks, see Numerical validation.

4. Features that are separate from recovery

Feature	Role
Initial guesses	Existing model-specific estimators choose starting parameters before solving; supplied guesses and fixed values retain their meaning.
Analytic / finite-difference derivatives	Determine how the solver obtains slopes. Custom equations use finite differences; analytic built-ins and the finite-difference configuration have separate regression coverage.
Absolute / Nominal parameter uncertainty	Control reported covariance scaling and interpretation. They do not select a different recovery objective.
Fixed parameters and limits	Constrain estimates throughout solving and recovery. A parameter at a limit can lack a symmetric uncertainty interval.
All parameters fixed	Computes projections and diagnostics without estimating coefficients; measured x+y precision checks still apply.
Weighted Mean	Uses its direct statistical formula; its reports and plots share the normal presentation paths.
Reusable saved solution	Retains its recorded numerical result and provenance. Viewing a saved fit does not request a fresh optimization.

The numerical engine uses ODRPACK95. A converged status is an optimization stopping condition, not proof of model suitability, justified measurement errors, identifiable parameters or a global minimum. Interpret the fitted curve, residuals, parameter uncertainties and warnings together.