It is important when evaluating diagnostic test accuracy to include the entire spectrum of disease which is normally found in the population being tested. For Johne's disease this should include disease-free populations of animals for specificity analysis and infected populations of animals, composed of both those shedding M. paratuberculosis in their feces and those not shedding, for sensitivity analysis. Data from the National Repository for Paratuberculosis specimens will be presented to illustrate how the choice of samples for test sensitivity and specificity analysis can influence test accuracy determination. For example, if sensitivity analysis was done using only culture-positive cows and specificity analysis was done on animals in the same herd but having a minimum of three negative fecal cultures over a 24 month period, the sensitivity and specificity of the standard complement fixation (CF) test used in the United States (greater than or equal to 1:8 = positive) would have been 54.6% and 89.2%, respectively. In reality, the CF test had a sensitivity and specificity of 38.4% and 99.0%, respectively, based on biopsy or necropsy and isolation of M. paratuberculosis from tissue specimens. Similar findings were found for all the diagnostic tests examined and points out the need to critically evaluate the populations of animals used to determine the sensitivity and specificity of a diagnostic test. If this is not done, erroneous conclusions will be made about the accuracy of diagnostic tests for paratuberculosis, particularly for tests not based on organism detection.