BYU

Abstract by Tyler Moncur

Personal Infomation


Presenter's Name

Tyler Moncur

Degree Level

Masters

Co-Authors

Michael Nelson

Abstract Infomation


Department

Mathematics

Faculty Advisor

Tyler Jarvis

Title

Hyperparameters in Dense Neural Networks

Abstract

Neural networks are extremely versatile, in effect being universal function approximators, but it can be extremely challenging to find the right set of parameters and hyperparameters. Model training is both expensive and difficult due to the large number of parameters and sensitivity to hyperparameters such as learning rate and architecture. Hyperparameter searches are notorious for requiring tremendous amounts of processing power and human resources. We show how to compute optimal initializations and learning rates for fully connected neural networks that use the Crelu activation function, which can reduce the cost of model training and hyperparameter searches significantly.