Our face is a key modality to convey emotions and infer intention. This makes face analysis an important factor in understanding the underlying mechanisms of interaction. Automatic solutions for facial expression recognition promise to deliver a significant fraction of the currently missing component of non-verbal communication to the human-machine interaction enabling more fulfilling experience closely modelling interpersonal communication. This thesis presents three major contributions aimed to overcome a number of issues currently preventing modern face analysis solutions from being applied in practice. The problem of reliable automatic discovery of facial actions is first considered from the point of view of manual feature craft, exploring ways to highlight features related to interpersonal commonalities in facial expression appearance, disregarding those corresponding to environmental conditions and subjective differences. It is then approached from the Multi-Task and Transfer learning perspective, presenting solutions for cost and performance efficient training of facial expression detection algorithms. Finally, a novel solution is proposed for multi-database heterogeneous data representation aimed to provide an environment for better generalisable face analysis solutions training and evaluation.