In this article, we review recent evidence of infants' early competence in perceiving and interpreting the actions of others. We present a theoretical model that decomposes the timeline of action perception into a series of distinct processes that occur in a particular order. Once an agent is detected, covert attention can be allocated to the future state of the agent (priming), which may lead to overt gaze shifts that predict goals (prediction). Once these goals are achieved, the consequence of the agents' actions and the manner in which the actions were performed can be evaluated (evaluation). We propose that all of these processes have unique requirements, both in terms of timing and cognitive resources. To understand more fully the rich social world of infants, we need to pay more attention to the temporal structure of social perception and ask what information is available to infants and how this changes over time.