Sensor calibration fit tool

How it works

A sensor is calibrated by measuring its output at known inputs and fitting a curve that turns later readings back into an input. This tool fits an end-point line, a least squares line, a polynomial or a two variable curve with temperature terms to pasted calibration data, and reports the coefficients, the residuals, the maximum error, the non-linearity, the hysteresis and the repeatability.

1. Choose the model. The end-point line joins the mean outputs at the lowest and highest inputs. The best-fit line and the polynomials minimise the sum of squared residuals. The two variable fit adds terms in the temperature difference from a reference temperature, so the offset and the sensitivity can drift with temperature.line: output = a0 + a1 x; polynomial: output = a0 + a1 x + a2 x^2 + a3 x^3; two variables: output = a0 + a1 x + a2 x^2 + b1 (T − Tref) + c1 x (T − Tref)
2. Solve by least squares. Each term is a column of a design matrix A and the outputs form a vector y. The coefficients minimise the squared residual. The tool solves it by Householder QR factorisation after scaling every column to unit length, which stays accurate for the small orders used and avoids the loss of precision that the normal equations suffer when the columns are nearly dependent.minimise | A c − y |^2; A = Q R; R c = Q^T y; standard error of a coefficient = s x sqrt( [ (A^T A)^−1 ]jj ), s^2 = SSres / (n − p)
3. Residuals and error. The residual of each point is the measured output minus the fitted output. The maximum error is the largest residual size, given in output units, as a percentage of the full scale output and referred to the input by dividing by the local slope. R squared compares the residual with the spread of the outputs about their mean.residual = y − fit(x, T); error % FS = 100 x max |residual| / FS; R^2 = 1 − SSres / SStot
4. Non-linearity, hysteresis and repeatability. Non-linearity is the largest distance of the mean output at each input from the least squares straight line through those means, using readings corrected to the reference temperature where temperature terms are fitted. Hysteresis is the largest difference between the mean up sweep and the mean down sweep at the same input. Repeatability is the spread of repeated readings at the same input, direction and temperature, given as a range and as a pooled standard deviation.non-linearity % FS = 100 x max |mean(y) − line(x)| / FS; hysteresis % FS = 100 x max |mean(up) − mean(down)| / FS
5. Use the calibration in reverse. In service the sensor gives a reading and the input is wanted. The tool finds the input within the calibrated range at which the fitted curve equals the reading, at the temperature you give for the two variable fit. A reading outside the range returns no input rather than an extrapolation.find x with fit(x, T) = reading

Sources

The methods and default figures rest on these public sources. The wording here is our own; read the source for its full text.

• NIST/SEMATECH e-Handbook of Statistical Methods, section 4.1.4.1, Linear Least Squares Regression (opens in a new tab), National Institute of Standards and Technology, Online edition, accessed 25 October 2026. Least squares fitting of models that are linear in the parameters, including polynomials.
• NIST/SEMATECH e-Handbook of Statistical Methods, section 2.3.6, Instrument calibration over a regime (opens in a new tab), National Institute of Standards and Technology, Online edition, accessed 25 October 2026. Calibration curves fitted by least squares over a range, and their use in reverse to calibrate later readings.