Even though cancer survivors are often able to continue working after they have been diagnosed, they may have health impairments resulting in reduced work ability. We studied the current work ability of 591 employed people with an early-stage of breast cancer, lymphoma, testicular or prostate cancer, and 757 referents. We also investigated whether the survivors perceived that cancer had impaired their work ability, and which disease-related, socio-demographic and social factors at work had an impact on their work ability. The work ability of the cancer survivors did not differ from that of their referents. Among the survivors, 26% reported that their physical work ability, and 19% that their mental work ability had deteriorated due to cancer. The survivors who had other diseases or had had chemotherapy, most often reported impaired work ability, whereas survivors with a strong commitment to their work organisation, or a good social climate at work, reported impairment less frequently.