Internationally accepted exposure-response relationships show that railway noise causes less annoyance than road traffic and aircraft noise. Railway transport, both passenger and freight transport, is increasing, and new railway lines are planned for environmental reasons. The combination of more frequent railway traffic and faster and heavier trains will, most probably, lead to more disturbances from railway traffic in the near future. To effectively plan for mitigations against noise and vibration from railway traffic, new studies are needed to obtain a better basis of knowledge. The main objectives of the present study was to investigate how the relationship between noise levels from railway traffic and general annoyance is influenced by (i) number of trains, (ii) the presence of ground borne vibrations, and (iii) building situational factors, such as orientation of balcony/patio and bedroom window. Socio-acoustic field studies were executed in residential areas; (1) with relatively intense railway traffic; (2) with strong vibrations, and; (3) with the most intense railway traffic in the country. Data was obtained for 1695 respondents exposed to sound levels ranging from L(Aeq,24h) 45 to 65 dB. Both number of trains and presence of ground-borne vibrations, and not just the noise level per se, are of relevance for how annoying railway noise is perceived. The results imply that, for the proportion annoyed to be equal, a 5 - 7 dB lower noise level is needed in areas where the railway traffic causes strong ground-borne vibrations and in areas with a very large number of trains. General noise annoyance was twice as high among residents in dwellings with balcony / patio oriented towards the railway and about 1.5 times higher among residents with bedroom windows facing the railway.