The importance of between-group competition in the social evolution of animal societies is controversial, particularly with respect to understanding the origins and maintenance of cooperation in our own species. Among primates, aggressive between-group encounters are often rare or strikingly absent, a phenomenon that in some species has been ascribed to the presence of collective action problems. Here, we report on a series of comparative tests that show that the intensity of between-group contest competition is indeed lower in species that experience a collective action problem while controlling for predictions from an “ideal gas” model of animal encounters and general species’ ecology. Species that do not succumb to the collective action problem are either cooperative breeders, are characterized by philopatry of the dominant sex, or live in relatively small groups with few individuals of this dominant sex. This implies that collective action problems are averted either through shared genes and benefits or a by-product mutualism in which the territorial behavior of some privileged individuals is not affected by the behavior of others. We conclude that across the primate taxon, the intensity of between-group competition is predominantly constrained by a social dilemma among group members, rather than ecological conditions, and that the collective action problem is thus an important selective pressure in the evolution of primate (including human) cooperation and sociality.