Expression of the hyaluronan-mediated motility receptor (RHAMM, CD168) predicts adverse clinicopathological features and decreased survival for colorectal cancer (CRC) patients. Using full tissue sections, we investigated the expression of RHAMM in tumor budding cells of 103 primary CRCs to characterize the biological processes driving single-cell invasion and early metastatic dissemination. RHAMM expression in tumor buds was analyzed with clinicopathological data, molecular features and survival. Tumor budding cells at the invasive front of CRC expressed RHAMM in 68% of cases. Detection of RHAMM-positive tumor budding cells was significantly associated with poor survival outcome (P = .0312), independent of TNM stage and adjuvant therapy in multivariate analysis (P = .0201). RHAMM-positive tumor buds were associated with frequent lymphatic invasion (P = .0007), higher tumor grade (P = .0296), and nodal metastasis (P = .0364). Importantly, the prognostic impact of RHAMM expression in tumor buds was maintained independently of the number of tumor buds found in an individual case (P = .0246). No impact of KRAS/BRAF mutation, mismatch repair deficiency and CpG island methylation was observed. RHAMM expression identifies an aggressive subpopulation of tumor budding cells and is an independent adverse prognostic factor for CRC patients. These data support ongoing efforts to develop RHAMM as a target for precision therapy.