The number of male gametes produced is critical for reproductive success and varies greatly between and within species. Evolutionary reduction of male gamete production has been widely reported in plants as a hallmark of the selfing syndrome, as well as in humans. Such a reduction may simply represent deleterious decay, but evolutionary theory predicts that breeding systems could act as a major selective force on male gamete number: while large numbers of sperm should be produced in highly promiscuous species because of male-male gamete competition, reduced sperm numbers may be advantageous at lower outcrossing rates because of the cost of gamete production. Here we used genome-wide association study (GWAS) to show a signature of polygenic selection on pollen number in the predominantly selfing plant Arabidopsis thaliana. The top associations with pollen number were significantly more strongly enriched for signatures of selection than those for ovule number and 107 phenotypes analyzed previously, indicating polygenic selection. Underlying the strongest association, responsible for 20% of total pollen number variation, we identified the gene REDUCED POLLEN NUMBER 1 affecting cell proliferation in the male germ line. We validated its subtle but causal allelic effects using a quantitative complementation test with CRISPR-Cas9-generated null mutants in a nonstandard wild accession. Our results support polygenic adaptation underlying reduced male gamete numbers.