Previous studies suggest physical activity improves cognition and lowers Alzheimer's disease (AD) risk. However, key AD pathogenic factors that are thought to be influenced by physical activity, particularly plasma amyloid-β (Aβ) and Aβ brain load, have yet to be thoroughly investigated. The objective of this study was to determine if plasma Aβ and amyloid brain deposition are associated with physical activity levels, and whether these associations differed between carriers and non-carriers of the apolipoprotein E (APOE) ε4 allele. Five-hundred and forty six cognitively intact participants (aged 60-95 years) from the Australian Imaging, Biomarkers and Lifestyle Study of Ageing (AIBL) were included in these analyses. Habitual physical activity levels were measured using the International Physical Activity Questionnaire (IPAQ). Serum insulin, glucose, cholesterol and plasma Aβ levels were measured in fasting blood samples. A subgroup (n=116) underwent (11)C-Pittsburgh compound B (PiB) positron emission tomography (PET) scanning to quantify brain amyloid load. Higher levels of physical activity were associated with higher high density lipoprotein (HDL) (P=0.037), and lower insulin (P<0.001), triglycerides (P=0.019) and Aβ1-42/1-40 ratio (P=0.001). After stratification of the cohort based on APOE ε4 allele carriage, it was evident that only non-carriers received the benefit of reduced plasma Aβ from physical activity. Conversely, lower levels of PiB SUVR (standardised uptake value ratio) were observed in higher exercising APOE ε4 carriers. Lower plasma Aβ1-42/1-40 and brain amyloid was observed in those reporting higher levels of physical activity, consistent with the hypothesis that physical activity may be involved in the modulation of pathogenic changes associated with AD.