In this paper we study the coevolutionary dynamics of knowledge creation, diffusion and the formation of R&D collaboration networks. Differently to previous works, knowledge is not treated as an abstract scalar variable but represented by a portfolio of ideas that changes over time through innovations and knowledge spillovers between collaborating firms. The collaborations between firms, in turn, are dynamically adjusted based on the firms' expectations of learning a new technology from their collaboration partners. We analyze the behavior of this dynamic process and its convergence to a stationary state, in relation to the rates at which innovations and costly R&D collaboration opportunities arrive, and the rate of creative destruction leading to the obsolescence of existing tech- nologies. We quantify the innovation gains from collaborations, and show that there exists a critical level for the technology learning success probability in collaborations below which an economy with weak in-house R&D capabilities does not innovate even in the presence of R&D collaborations. More- over, we show that the interplay between knowledge diffusion and network formation can give rise to a cyclical pattern in the collaboration intensity, which can be described as a damped oscillation. We confirm this novel observation using an empirical sample of a large R&D collaboration network over the years 1985 to 2011. We then study the efficient network structure, compare it to the decentralized equilibrium structures generated, and design an optimal network policy to maximize welfare in the economy. Our efficiency analysis further allows us to study the effect of competition on innovation in R&D intensive industries where R&D collaborations between firms are commonly observed.