SECOND ORDER ALGORITHM FOR SPARSELY CONNECTED NEURAL NETWORKS

Author

Parastoo Kheirkhah

Graduation Semester and Year

2016

Language

English

Document Type

Thesis

Degree Name

Master of Science in Electrical Engineering

Department

Electrical Engineering

First Advisor

Michael T Manry

Abstract

A systematic two-step batch approach for constructing a sparse neural network is presented. Unlike other sparse neural networks, the proposed paradigm uses orthogonal least squares (OLS) to train the network. OLS based pruning is proposed to induce sparsity in the network. Based on the usefulness of the basic functions in the hidden units, the weights connecting the output to hidden units and output to input units are modified to form a sparse neural network. The proposed hybrid training algorithm has been compared with the fully connected MLP and sparse softmax classifier that uses second order training algorithm. The simulation results show that the proposed algorithm has significant improvement in terms of convergence speed, network size, generalization and ease of training over fully connected MLP. Analysis of the proposed training algorithm on various linear and non-linear data files is carried out. The ability of the proposed algorithm is further substantiated by clearly differentiating two separate data sets when feed into the proposed algorithm. The experimental results are reported using 10-fold cross validation. Inducing sparsity into a fully connected neural network, pruning of the hidden units, Newton’s method for optimization, and orthogonal least squares are the subject matter of the present work.

Keywords

Neural networks, Sparsity, Second order algorithm, Orthogonal least square, Hessian matrix

Disciplines

Electrical and Computer Engineering | Engineering

License

This work is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 4.0 International License.

Comments

Degree granted by The University of Texas at Arlington

Recommended Citation

Kheirkhah, Parastoo, "SECOND ORDER ALGORITHM FOR SPARSELY CONNECTED NEURAL NETWORKS" (2016). Electrical Engineering Theses. 358.
https://mavmatrix.uta.edu/electricaleng_theses/358