Remnant planetesimals might have played an important role in reducing the orbital eccentricities of the terrestrial planets after their formation via giant impacts. However, the population and the size distribution of remnant planetesimals during and after the giant impact stage are unknown, because simulations of planetary accretion in the runaway growth and giant impact stages have been conducted independently. Here we report results of direct N-body simulations of the formation of terrestrial planets beginning with a compact planetesimal disk. The initial planetesimal disk has a total mass and angular momentum as observed for the terrestrial planets, and we vary the width (0.3 and 0.5 AU) and the number of planetesimals (1000-5000). This initial configuration generally gives rise to three final planets of similar size, and sometimes a fourth small planet forms near the location of Mars. Since a sufficient number of planetesimals remains, even after the giant impact phase, the final orbital eccentricities are as small as those of the Earth and Venus.