Many important physical applications are governed by the wave equation. The formulation as time domain boundary integral equations involves retarded potentials. For the numerical solution of this problem, we employ the convolution quadrature method for the discretization in time and the Galerkin boundary element method for the space discretization. We introduce a simple a priori cut-off strategy where small entries of the system matrices are replaced by zero. The threshold for the cut-off is determined by an a priori analysis which will be developed in this paper. This analysis will also allow to estimate the effect of additional perturbations such as panel clustering and numerical integration on the overall discretization error. This method reduces the storage complexity for time domain integral equations from O(M2N) to O(M2N1/2 logM), where N denotes the number of time steps and M is the dimension of the boundary element space.