Technical analyses of several recent power blackouts revealed that a group of generators going out-of-step with the rest of the power system is often a precursor of a complete system collapse. Out-of-step protection is designed to assess the stability of the evolving swing after a disturbance and take control action accordingly. However, the settings of out-of-step relays are found to be unsatisfactory due to the fact that the electromechanical swings that occurred during relay commissioning are different in practice. These concerns motivated the development of a novel approach to recalculate the out-of-step protection settings to suit the prevalent operating condition. With phasor measurement unit (PMU) technology, it is possible to adjust the setting of out-of-step relay in real-time. The setting of out-of-step relay is primarily determined by three dynamic parameters: direct axis transient reactance, quadrature axis speed voltage and generator inertia. In a complex power network, these parameters are the dynamic parameters of an equivalent model of a coherent group of generators. Hence, it is essential to identify the coherent group of generators and estimate the dynamic model parameters of each generator in the system first in order to form the dynamic model equivalent in the system. The work presented in this thesis develops a measurement-based technique to identify the coherent areas of power system network by analysing the measured data obtained from the system. The method is based on multivariate analysis of the signals, using independent component analysis (ICA). Also, a technique for estimating the dynamic model parameters of the generators in the system has been developed. The dynamic model parameters of synchronous generators are estimated by processing the PMU measurements using unscented Kalman filter (UKF).