Just as NAND gates are not unique (NOR gates are also universal), nor is any particular neuron implementation—indeed, any generic smooth nonlinear activation function is universal [8, 9]. The class of functions that can be exactly expressed by a neural network must be invariant under ...