In a longitudinal population-based ultrasound survey, we evaluated the reproducibility of carotid plaque detection, off-line vs. online visual classification of plaque echogenicity and computer-assisted plaque echogenicity (grey-scale median, GSM) classification and plaque area measurements. The number of paired observations in the reproducibility analyses was 107 in the baseline study and 83 in the follow-up study. In addition, 198 and 222 images were selected from the baseline and the follow-up study for GSM- and plaque-area analyses. The total number of plaque images (11,160) was used to obtain comparative reference values. Despite good agreement in the reproducibility study (kappa values ranging from 0.52 to 0.57), there was a substantial drift in online visual classification of plaque echogenicity during the survey period. Inter- and intraobserver agreement on computer-assisted GSM classification was substantial, with kappa values (95% CI) of 0.77 (0.73 to 0.80) and 0.79 (0.75 to 0.84), respectively. A systematic bias in plaque area measurements was observed. Visual online classification may introduce systematic measurement errors that are not intercepted in a reproducibility study of restricted duration. Computer-assisted off-line classification had better reproducibility. However, the method is influenced by measurement errors, both in the outlining of the plaque and in the standardization procedure.