Bayesovské učení binárních neuronových sítí

Abstract

Neural Networks with binary weights are of special interest as they are computation friendly and do not demand a lot hardware. However, training is rather challenging as they these binary weights do not have a gradient. Bayesian learning averages over a range of models that fit the data well and is thus able to provide us with a decent model itself. Another advantage Bayesian learning posses over existing common methods for training binary neural networks is that it is not empirical. However, despite these advantages over their continuous counter parts, Binary Neural Networks are significantly lag behind traditional neural networks in terms of performance. This is because training Binary networks is particularly difficult as it involves a discrete optimization problem. Moreover, traditional methods such Stochastic Gradient Descent can not be used to update the weights as the they are discrete and do not have a gradient. In this thesis, various Bayesian methods were explored such as Variational Bayesian Learning and Maximum Likelihood. Their performance is analyzed on a toy dataset drawn from generative data model. Existing methodology is also reviewed.Neural Networks with binary weights are of special interest as they are computation friendly and do not demand a lot hardware. However, training is rather challenging as they these binary weights do not have a gradient. Bayesian learning averages over a range of models that fit the data well and is thus able to provide us with a decent model itself. Another advantage Bayesian learning posses over existing common methods for training binary neural networks is that it is not empirical. However, despite these advantages over their continuous counter parts, Binary Neural Networks are significantly lag behind traditional neural networks in terms of performance. This is because training Binary networks is particularly difficult as it involves a discrete optimization problem. Moreover, traditional methods such Stochastic Gradient Descent can not be used to update the weights as the they are discrete and do not have a gradient. In this thesis, various Bayesian methods were explored such as Variational Bayesian Learning and Maximum Likelihood. Their performance is analyzed on a toy dataset drawn from generative data model. Existing methodology is also reviewed.