While post-stroke dementia has been extensively investigated, the large number of patients with mild cognitive impairment (MCI) following stroke has received less attention, and reports on the longitudinal course of such impairment are inconsistent in their findings. We examined patients with MCI (n = 45) or no cognitive impairment (NCI) (n = 59), based on consensus criteria following detailed neuropsychological assessments and magnetic resonance imaging (MRI) scans, and compared them with healthy control subjects (n = 84), all of whom were assessed at two time points, 3 years apart. The MCI at baseline in this group was judged to be vascular in etiology (vaMCI). Incident dementia was diagnosed in 24.4% of vaMCI and 8.5% of NCI subjects and no control subjects over 3 years, giving a rate of conversion of approximately 8% per year in post-stroke vaMCI. The vaMCI group showed greater decline in logical memory than the NCI group. Within the vaMCI group, those who developed dementia had great decline in language and executive function. Compared with NCI patients, those with vaMCI had more vascular risk factors and more white matter hyperintensities on MRI at baseline, but did not differ in their brain or hippocampal volumes. Neither MRI volumetric measures nor interval cerebrovascular events predicted decline in function. The major determinant of decline and categorical transition was impaired performance at baseline, suggesting that those with mild impairment post-stroke are more vulnerable to subsequent decline.