The current interpretation of Artificial Neural Networks (ANNs) has several shortcomings, problems including time dependency between previous (in time) outputs and current inputs (FSM like behaviour), solving more than one problem in the same network and interference (catastrophic forgetting) the network forget previous trained problems when exposed to a new problem. Developmental neural networks may be a method toward new network architectures and training regimes that can expand what problem ANNs can efficiently solve. In developmental AANs the network is grown or developed in a bio-inspired fashion. The network and weights emerged out of an emerging structure governed by rules for cell replication, cell differentiation, cell death, axion growth, axion change and axion death. In such a network the finalized network structure is a product of self-organization in combination with environmental influence (learning).
This project will be in collaboration with Prof. Julian Miller University of York (UK). Julian has developed a software model for evolution and development of ANNs with a specific target of solving multiple problems (in c++). That is a network that can start with one problem and be expanded to solve more problems without forgetting the earlier problems. Julian’s software and model will be available and used in the project. However, in this project I would like to address an observed side effect of Julian’s networks. The developed network seems to share neurons between problems, further the networks are quite small. As such,the size and structure of the network’s influence on learning. More specific; explore the “learning landscape” of developed ANNs compared to standard feedforward methods., e.g. backpropagation.
Specific problems that can be address in this project:
-Is it possible to train a developed ANN using standard methods, e.g. backprop?
-If possible at all, is such a developmental network more or less resource demanding to train?
-Way and how does the network properties influence on the search space for learning?
-And many more interesting problems as we know more…..
(It is possible to follow up by asking the same questions for ANNs that are binarized ) (in this project or as a seperat project if your intrests are in that direction, just contact me and we can formalize a project description).
Some more information:
Last years ALife workshop on Developmental Neural Networks: https://www.irit.fr/devonn/files/wilson_2019_presentation.pdf
Paper by Julian Miller: “Evolving developmental programs that build neural networks for solving multiple problems” (https://static.wixstatic.com/ugd/5ef763_e10e2ada742b42219df40aa9e046598d.pdf)
“ Evolving programs to build artificial neural networks” (https://static.wixstatic.com/ugd/5ef763_976e56fc945244278a246d14dc3f54a8.pdf)
Background
ANNs was originally proposed as a model for how the brain processes information. The concept emerged as a way to compute and as an abstraction to understand the brain.
ANNs are traditionally defined as a fixed structure. Neurons are spatially defined as part of the Input layer, hidden layers or output layer connected by directed weighted edges. The neurons compute a non-linear function of the sum of its inputs, the edges are a linear transformation (weighted) from neuron output to neuron input. The functionality of the network is a product of the set of weights (and neuron function) present in the network. Generally a feedforward network.
The weights for a specific functionality is found by some training algorithm, There is some defined scheme for adjusting the weights based on a performance measurement of the network. Adjustment is based on the performance on the network on some test at current time (t), the network is adjusted and tested again at t + 1, and so on. Currently (in most schemes) the weight adjustment is done weight by weight, there is no dependency between the individual weight adjustments.
Even though ANNs now is top of the pops once more with deep learning, convolutional neural networks and all the other buzz around there are few fundamental new principles and methods, i.e. the network architecture is a fixed structure forming a feedforward network, using some training process based on global information.
(If you are interested in any of my project please contact me for a short informal chat)
[ Skjul beskrivelse ]
