We show how gold recrystallizes when under the influence of electrochemical potentials. This "cold annealing" occurs without charge transfer reactions and preserves nanoscale structural features. By performing the process on plasmonic nanostructures, grain growth is monitored noninvasively by optical spectroscopy. In this way, the influence from crystal structure on plasmon resonances can be investigated independently. Observed spectral changes are in excellent agreement with analytical models and changes in electron relaxation time and plasma frequency are calculated.