The characteristics of recycled and secondary aggregates: coarse recycled concrete aggregate (RCA), fine glass cullet aggregate (GCA) and fine copper slag aggregate (CSA), and their effects on concrete deformation properties: elastic modulus, creep and shrinkage, have been studied. A novel Analytical Systemisation method was developed for the analysis and evaluation of the results sourced from 713 studies, undertaken by 960 authors from 537 institutions in 46 countries during 1972–2017, forming a data matrix having over 400,000 data points. Aggregate physical properties were found to be affected by the crushing process, more so for RCA than GCA and CSA. It was found that RCA reduces the resistance of concrete to deformation, whilst GCA and CSA result in no change or an improvement. The change in the deformation was shown to be affected by aggregate content, concrete strength and other factors. Most of the existing models were found not to consider the aggregate effect in estimating the deformation of concrete. Three new empirical models, essentially based on aggregate stiffness in the form of aggregate absorption, aggregate content and its ratio to cement content, have been developed for estimating the deformation of concrete made with aggregate suitable for use in structural concrete.