The health benefits of physical activity (PA) have been well documented. However, there is less research investigating whether or not these health benefits might differ among males and females or among subjects characterized by different levels of body mass index (BMI), waist-to-hip ratio (WHR), and waist circumference (WC). Baseline total PA, BMI, WHR and waist circumference were measured in 14,585 men and 26,144 women who participated in the Swedish National March. Their effects on all-cause mortality were analyzed with a follow-up time of almost 10 years. Sedentary men with a BMI ≥ 30 had a 98% (95% CI: 30-201%) increased risk of mortality compared to normal weight men with a high level of total PA. The same trend was observed for sedentary men with high WHR or waist circumference, compared to lean and highly active men. Sedentary women with a waist circumference of 88 cm or more had almost doubled, i.e. 97% (95% CI: 35-189%) increased mortality risk compared to physically active women with a waist circumference below 80 cm. BMI in men, but waist circumference in women better forecast all-cause mortality. We found no substantial effect modification between different measures of adiposity and physical activity-physical inactivity and obesity seem to increase total mortality risk independently and additively.