How do verbal descriptions affect visual memory over the short and long term? Here we show for the first time that verbal labeling can boost visual memories, but the source of this benefit depends on whether representations are maintained over the short term in visual working memory or over the long term in visual long-term memory. Across three experiments, we contrasted color memory of randomly colored objects when participants labeled (a) the color, (b) the object, or (c) the color-object binding, to memory under an articulatory suppression condition inhibiting labeling. Memory was tested at two time points: after three objects (visual working memory) and at the end of the experiment (visual long-term memory). In Experiment 1, color labeling improved, whereas object labeling impaired, visual working memory in comparison to suppression. Visual long-term memory remained unchanged across conditions. Experiment 2 tested whether this was attributable to poor overall long-term learning by repeating the colored objects over three successive working memory trials. This increased performance over the short and long term, yet labeling did not change learning rate over repetitions or delayed memory performance, showing no long-term memory benefit. In Experiment 3, a labeling benefit was observed when the color-object binding was labeled both over the short and long term. Mixture modeling indicated that color-labeling benefits in visual working memory resulted from an increase of detailed visual memory, whereas long-term memory benefits accrued from categorical representations. Our findings point to dissociations on the role of language in visual working memory and visual long-term memory.