Assessing the contributions of technical error and biological variability to error of measurement in reliability studies

A neglected issue in reliability studies is the contribution of technical (device) error and biological variability to the standard (typical) error of measurement. These contributions can be estimated with mixed modeling when subjects are measured simultaneously with two devices on two occasions. The aim of this study was to develop and validate a spreadsheet-based method to separately estimate technical error and biological variability, and to compare its performance with a mixed model. Change scores between occasions and devices are combined to yield standard deviations (SDs) for technical errors and biological variability in raw, percent, and standardized units, with bootstrapped confidence limits. The analyses were reproduced in a computer program and evaluated by simulation for true values of means and SDs mimicking real data with sample sizes of up to 50. Bias (deviation from true values) and uncertainty (width of confidence intervals) in the means and SDs were assessed by standardization, and coverage of true values by confidence intervals was assessed as nominal, under or over. Empirical correction factors were derived to correct small-sample bias in the SDs and eliminate under-coverage. Spreadsheet estimates were then compared to those of a mixed model. Bias was trivial for all estimates. Uncertainty was similar for the spreadsheet and mixed model, although substantial even with the largest sample size. The spreadsheet generally outperformed the mixed model for coverage. Analysis of jump-height measurements of 31 young adults recorded simultaneously with a photoelectric device and a jump mat provided good evidence that the technical errors produced respectively trivial and substantial increases in the typical error. In conclusion, the spreadsheet is a trustworthy tool for evaluating device error.