This thesis presents a theoretical investigation of the application of advanced modelling formats in high-speed fibre lightwave systems. The first part of this work focuses on numerical optimisation of dense wavelength division multiplexing (DWDM) system design. We employ advanced spectral domain filtering techniques and carrier pulse reshaping. We then apply these optimisation methods to investigate spectral and temporal domain characteristics of advanced modulation formats in fibre optic telecommunication systems. Next we investigate numerical methods used in detecting and measuring the system performance of advanced modulation formats. We then numerically study the combination of return-to-zero differential phase-shift keying (RZ-DPSK) with advanced photonic devices. Finally we analyse the dispersion management of Nx40 Gbit/s RZ-DPSK transmission applied to a commercial terrestrial lightwave system.