Advances in the understanding of the molecular basis of diseases have expanded the number of plausible therapeutic targets for the development of innovative agents in recent decades. However, although investment in pharmaceutical research and development (R&D) has increased substantially in this time, the lack of a corresponding increase in the output in terms of new drugs being approved indicates that therapeutic innovation has become more challenging. Here, using a large database that contains information on R&D projects for more than 28,000 compounds investigated since 1990, we examine the decline of R&D productivity in pharmaceuticals in the past two decades and its determinants. We show that this decline is associated with an increasing concentration of R&D investments in areas in which the risk of failure is high, which correspond to unmet therapeutic needs and unexploited biological mechanisms. We also investigate the potential variations in productivity with regard to the regional location of companies and find that although companies based in the United States and Europe differ in the composition of their R&D portfolios, there is no evidence of any productivity gap.