Specific blue-yellow color vision impairment has been reported in cocaine-dependent users and it was postulated that drug-induced changes in retinal dopamine neurotransmission are responsible. However, it is unclear whether these changes are confined to chronic cocaine users, whether they are specific for dopaminergic stimulants such as cocaine and amphetamine, and whether they are related to cognitive functions such as working memory, encoding, and consolidation. In 47 occasional and 29 dependent cocaine users, 23 MDMA users, and 47 stimulant-naïve controls, color vision discrimination was measured with the Lanthony Desaturated Panel D-15 Test and memory performance with the Auditory Verbal Learning Test. Both occasional and dependent cocaine users showed higher Color Confusion Indices than controls. Users of the serotonergic stimulant MDMA (26%), occasional (30%) and dependent cocaine users (34%) exhibited more frequent blue-yellow color vision disorders compared to controls (9%). Inferior performance of MDMA users was caused by a subgroup with high amphetamine co-use (55%), while MDMA use alone was not associated with decreased blue-yellow discrimination (0%). Cognitive performance was worse in cocaine users with color vision disorder compared to users and controls with intact color vision and both color vision impairment and cognitive deficits were related to cocaine use. Already occasional cocaine and amphetamine use might induce blue-yellow color vision impairment, whereas the serotonergic stimulant MDMA does not impair color vision. The association between color vision impairment and cognitive deficits in cocaine users may reflect that retinal and cerebral dopamine alterations are linked to a certain degree.