In this thesis, we introduce two different methods for determining noise covariance matrices in order to improve the stability and accuracy in state estimation and output prediction of discrete-time linear time varying (LTV) and nonlinear state space systems. The first method is based on the auto-covariance least squares (ALS) method, where the noise covariance matrices can be estimated by establishing a linear relationship between noise covariances and correlations of innovation sequence, hence solving a linear least squares problem. For LTV systems, we propose a new ALS algorithm that does not involve any approximations in the formulation. Our new ALS algorithm has fewer parameters to determine and can provide more accurate noise covariance estimation even when the historical output measurement window is not sufficiently long, comparing to an existing method. In addition to the noise covariance estimates, our ALS algorithm can also provide the estimate of the initial state error covariance, which is required by most state estimation methods. For higher-order systems, we also provide a much faster and less memory demanding formulation by splitting large Kronecker products with sums of smaller Kronecker or Schur products. For nonlinear systems, we have to approximate nonlinear parts as time-varying matrices by linearizing the nonlinear function around current state estimates. In addition to the extended Kalman Filter (EKF), our ALS algorithm also uses moving horizon estimation (MHE) to estimate the system state. MHE guarantees stability, is able to add state constraints and provides more accurate state estimates and local linearizations around the current state than the EKF. The second method is based on expectation maximization (EM), where the noise covariance matrices are determined by recursively maximizing the likelihood of covariance matrices, given output measurements. In our method, the noise covariance matrices are estimated using a semi-definite programming (SDP) solver, so that the results are more accurate and guaranteed to be positive definite. We propose a new EM algorithm that, combined with MHE and full information estimation (FIE) rather than a Kalman-based filter/smoother, allows the addition of state constraints, provides stable and more accurate estimates, so that the performance of noise covariance estimation can be significantly improved. Finally, we apply our noise covariance estimation methods to ocean wave prediction for the control of a wave energy converter (WEC), in order to approach optimal efficiency of wave energy extraction. We use a state space model representation for an autoregressive (AR) process, combined with noise covariance estimation, to simulate wave height forecasting based on data recorded at Galway Bay, Ireland. The simulation returns good wave predictions. Compared to existing wave prediction methods, our model has fewer parameters to tune and is able to provide more stable and accurate wave predictions by using a Kalman-based filter combined with the ALS or EM method.