Abstract by Tyler Moncur
Hyperparameters in Dense Neural Networks
Neural networks are extremely versatile, in effect being universal function approximators, but it can be extremely challenging to find the right set of parameters and hyperparameters. Model training is both expensive and difficult due to the large number of parameters and sensitivity to hyperparameters such as learning rate and architecture. Hyperparameter searches are notorious for requiring tremendous amounts of processing power and human resources. We show how to compute optimal initializations and learning rates for fully connected neural networks that use the Crelu activation function, which can reduce the cost of model training and hyperparameter searches significantly.