In this work, I will focus on ways in which we can build machine learning models that appropriately account for uncertainty, whether with computationally cheap estimates or with more expensive and reliable ones. In particular, I will explore how we can model distributions with Bayesian neural networks and how we can manipulate them depending on the task. The two main techniques for performing inference in Bayesian neural networks are variational inference and Markov chain Monte Carlo. I will look into the advantages and disadvantages of both methods and apply them to real-world problems. The emphasis is on how to achieve calibrated uncertainty estimates without compromising scalability. One contribution of this work is to offer a new method for implementing Bayesian neural networks within the framework of Bayesian decision theory, where Bayesian decision theory is important in all decision-making applications. A further contribution is developing sampling techniques that provide more reliable uncertainties, especially over data that lie outside the training distribution. Finally I also introduce a method for using Bayesian neural networks in an astrophysical application where it is vital that uncertainties are calibrated appropriately for the task.