Approximate Monte Carlo algorithms are not uncommon these days, their applicability is related to the possibility of controlling the computational cost by introducing some noise or approximation in the method. We focus on the stability properties of a particular approximate MCMC algorithm, which we term noisy Metropolis-Hastings. Such properties have been studied before in tandem with the pseudo-marginal algorithm, but under fairly strong assumptions. Here, we examine the noisy Metropolis-Hastings algorithm in more detail and explore possible corrective actions for reducing the introduced bias. In this respect, a novel approximate method is presented, motivated by the class of exact algorithms with randomised acceptance. We also discuss some applications and theoretical guarantees of this new approach.