The association between diet and colorectal cancer has been studied in depth for many decades, with equivocal results. It has been hypothesized that cancers arising in the distal and proximal colon have different pathologies, and therefore different risk factors. As such, it is possible that diet-related factors might influence colorectal neoplasia differently depending on the subsite. Recent evidence indicates that women may be more likely to develop proximal cancers than men. Additionally, the link between certain dietary factors and colorectal neoplasia in women seems to vary by menopausal status. Given these observations, women may be affected differently than men by diet-related factors. The objective of this article was therefore to review the data for diet and colorectal adenomas and cancer, and then attempt to address the potential differences in the association of diet-related factors and colorectal neoplasia in men and women. For total energy intake, selenium, and fiber, it seems that there may be slightly stronger effects in men as compared with women, whereas calcium and folate seem to affect both sexes similarly. With regard to vitamin D and colorectal cancer, women may exhibit stronger associations than men. Perhaps the most evidence for a sex-specific effect is observed for obesity, where more substantial direct relationships between body size and colorectal neoplasia have been reported for men than for women. However, this observation may be influenced by the differential effects in women by menopausal status. Further research on sex-specific dietary effects is warranted.