Bio-inspired Evolutionary Multi-reservoir Neural Network
- 3 minsIntro
Inspired by the modality independent but functionally connected brain regions, the concept of evolving ensemble neural network has been proposed and investigated in this project using reservoir computing (RC) paradigm. Specifically, we proposed an ensemble RC network model which is capable of automatically adapting and optimizing the synaptic and structural plasticity of a reservoir ensemble towards an optimal performance using the genetic algorithm. As shown in a real-life time series application - temperature prediction, the proposed model demonstrates superior performance over both the conventional single-reservoir model and the static reservoir ensemble model.
Related works were published in Structure Optimization of Dynamic Reservoir Ensemble Using Genetic Algorithm and Performance Optimization of Echo State Networks Through Principal Neuron Reinforcement.
Summary
- Investigated the feasibility and potential of multi-reservoir network.
- Explored bio-inspired evolutionary neural networks based on reservoir computing paradigm.
- Proposed dynamic multi-reservoir network leveraging structural evolution (with genetic algorithm) and synaptic plasticity (synaptic learning rules).
Motivaiton
Reservior computing (RC) is widely applied in dynamic system modeling and is especially famous for its ability of solving time dependent problems at relatively low computational expense. However, the underlying features of its random initialization and fixed topology also brings about some limitations: 1) it is not able to adapt itself over time to improve the performance, 2) given the fact that different topologies and internal connections of the reservoir may result in various dynamic behaviors, random initialization is far from competent, 3) limited by the inherent coupling among neurons within one reservoir, standard RC (networks with only one reservoir) could be less effective on some complex tasks that exhibit multiple sets of dynamics. So it leads to the natural idea of constructing an evolving ensemble RC network with multiple reservoirs.
The concept of this ensemble network is also inspired from the modality independent but functionally connected brain regions. Front lobe, parietal lobe, occipital lobe, temporal lobe, etc. are responsible for divers functions, like visual/auditory processing, memory, speech, movements and so on. Those regions, being connected together and working together, form a sophisticated dynamic system that can help us make complex decisions based on all kinds of information. We therefore try to translate such structure to the RC paradigm.
Assumptions
- Reservoir computing paradigm, to some extent, better mimics the topological properties of biological brains, where the connecting neurons do not exhibit explicit “layer structure”, but are rather interconnected in a “random like” loose recurrent manner.
- To better handle complex tasks that involve different information sources and divers responses, as its biological counterpart, the multi-reservoir network should be equiped with independent inputs and outputs. In other words, unlike fully-connected layers, different input/output neurons should have varied connection patterns and be connected with different hidden neurons.
- Multiple reservoirs can better propagate and process multiple sets of dynamics (arose by different kinds of inputs) than a single reservoir. The reservoirs should be loosely connected with each other to make use of the potential dependencies among various dynamics.
- The macro structural characteristics mainly contribute to functional relations (how well different information and regions may work together), and the micro synaptic characteristics mainly determines the elaborate degree of solving a task.
Evaluation Settings
- Database:
Performance
- Generative model performance:
- -