Studies investigating cognitive impairment in stroke-free patients with carotid artery stenosis have led to inconsistent results. Furthermore, the pathophysiological mechanism leading to cognitive impairment remains unclear. Cerebral hypoperfusion and arterio-arterial microembolization are discussed. The aims of our study were (1) to delineate patterns of cognitive impairment in stroke-free patients with carotid artery stenosis and (2) to investigate if cognitive impairment is independent of white matter lesion load in brain MRI. We identified 212 (93 women, mean age 70.2) stroke free, non-demented patients, who were referred for carotid artery stenting or because of subjective cognitive impairment. All patients completed a neurocognitive test battery measuring verbal fluency, constructional praxis, figural memory, verbal short-term- and long-term-memory, verbal recognition memory, semantic processing, speed of cognitive processing and divided attention. Grade of maximum carotid artery stenosis was categorized into three groups (mild, moderate, or severe). White matter lesion load was graded using a visual rating scale. Cognitive test scores of groups with different grades of carotid artery stenosis were compared. Univariate regression analysis was used to measure the predictive value of carotid artery stenosis. Multivariate logistic regression analysis was performed when integrating carotid artery stenosis and white matter lesion load. Carotid artery stenosis negatively correlated with measures of verbal fluency, constructional praxis, verbal short-term-memory, semantic processing, speed of cognitive processing, and divided attention. After adjustment for white matter lesions, carotid artery stenosis did not independently predict divided attention. Significance persisted in all other cognitive domains. In our selected group of patients, a higher grade of carotid artery stenosis is associated with cognitive decline. This process is independent of white matter lesion load. Possible pathophysiological implications are discussed.