We construct infinite dimensional symmetries of the Chalmers-Siegel action describing the self-dual sector of non-supersymmetric Yang-Mills. The symmetries are derived by virtue of a canonical transformation between the Yang-Mills fields and new fields that map the Chalmers-Siegel action to a free theory which has been used to construct a Lagrangian approach to the MHV rules. We describe the symmetries of the free theory in a quite general way which are an infinite dimensional algebra in the group algebra of isometries. We dimensionally reduce the symmetries of the action to write down symmetries of the Hitchin system and further, we extend the construction to the $N=4$ supersymmetric, self-dual theory. We review recent developments in the approach to calculating N=4 Yang-Mills scattering amplitudes using symmetry arguments. Super-conformal symmetry and the recently discovered dual super-conformal symmetry have been shown to be related as a Yangian algebra and moreover, anomalous terms appearing in their action on amplitudes lead to deformations of the generators which gives rise to recursive relationships between amplitudes.