Cell polarization is crucial for the functioning of all organisms. The cytoskeleton is central to the process but its role in symmetry breaking is poorly understood. We study cell polarization when fission yeast cells exit starvation. We show that the basis of polarity generation is de novo sterol biosynthesis, cell surface delivery of sterols, and their recruitment to the cell poles. This involves four phases occurring independent of the polarity factor cdc42p. Initially, multiple, randomly distributed sterol-rich membrane (SRM) domains form at the plasma membrane, independent of the cytoskeleton and cell growth. These domains provide platforms on which the growth and polarity machinery assembles. SRM domains are then polarized by the microtubule-dependent polarity factor tea1p, which prepares for monopolar growth initiation and later switching to bipolar growth. SRM polarization requires F-actin but not the F-actin organizing polarity factors for3p and bud6p. We conclude that SRMs are key to cell polarization.