The game of chess can be seen as a typical example for an expertise task requiring domain-specific training and experience. Despite intensive behavioural studies the neural underpinnings of chess performance and expertise are not entirely understood. A few functional neuroimaging studies have shown that expert chess players recruit different psychological functions and activate different brain areas while they are engaged in chess-related activities. Based on this functional literature, we predicted to find morphological differences in a network comprised by parietal and frontal areas and especially the occipito-temporal junction (OTJ), fusiform gyrus, and caudate nucleus. Twenty expert chess players and 20 control subjects were investigated using voxel-based and surface-based morphometry as well as diffusion tensor imaging. Grey matter volume and cortical thickness were reduced in chess players compared with those of control men in the OTJ and precunei. The volumes of both caudate nuclei were not different between groups, but correlated inversely with the years of chess playing experience. Mean diffusivity was increased in chess players compared with that of controls in the left superior longitudinal fasciculus and the Elo score (a chess tournament ranking) was inversely related to mean diffusivity within the right superior longitudinal fasciculus. To the best of our knowledge we showed for the first time that there are specific differences in grey and white matter morphology between chess players and control subjects in brain regions associated with cognitive functions important for playing chess. Whether these anatomical alterations are the cause or consequence of the intensive and long-term chess training and practice remains to be shown in future studies.