Neural Arithmetic

Abstract

Neural networks can learn complex functions, but they often have troubles with extrapolating even simple arithmetic operations on real numbers beyond the training range. A sub field of Neural Networks called Neural Arithmetic tries to address this extrapolation problem by making use of arithmetic operations like addition, multiplication, or division. This thesis provides a comparison between different arithmetic layers on their extrapolation performance for simple functions and on a recurrent task. Additionally, we exploit how arithmetic models can be used to build more transparent models by trying a simple equation discovery. The general introduction is done in Section 1. Section 2 describes Neural Networks and Neural Arithmetic layers. Section 3 contains the function learning, recurrent, and equation discovery experiments and Section 4 the conclusion.

Neural networks can learn complex functions, but they often have troubles with extrapolating even simple arithmetic operations on real numbers beyond the training range. A sub field of Neural Networks called Neural Arithmetic tries to address this extrapolation problem by making use of arithmetic operations like addition, multiplication, or division. This thesis provides a comparison between different arithmetic layers on their extrapolation performance for simple functions and on a recurrent task. Additionally, we exploit how arithmetic models can be used to build more transparent models by trying a simple equation discovery. The general introduction is done in Section 1. Section 2 describes Neural Networks and Neural Arithmetic layers. Section 3 contains the function learning, recurrent, and equation discovery experiments and Section 4 the conclusion.