A long-standing hypothesis suggests that the transition from hunting and gathering to agriculture results in people working harder, spending more time engaged in subsistence activities and having less leisure time1,2. However, tests of this hypothesis are obscured by comparing between populations that vary in ecology and social organization, as well as subsistence3,4,5,6. Here we test this hypothesis by examining adult time allocation among the Agta—a population of small-scale hunter-gatherers from the northern Philippines who are increasingly engaged in agriculture and other non-foraging work. We find that individuals in camps engaging more in non-foraging work spend more time involved in out-of-camp work and have substantially less leisure time. This difference is largely driven by changes in the time allocation of women, who spend substantially more time engaged in out-of-camp work in more agricultural camps. Our results support the hypothesis that hunting and gathering allows a significant amount of leisure time, and that this is lost as communities adopt small-scale agriculture.