Table of Contents

“博士后文库”序言
Preface
Chapter 1 Batch gradient method with smoothing L1/2 regularization for training of feedforward neural network 1
1.1 Introduction 1
1.2 Batch gradient method with smoothing L1/2 regularization 3
1.2.1 Batch gradient method with L1/2 regularization 3
1.2.2 Smoothing L1/2 regularization 5
1.3 Convergence results 6
1.4 Numerical examples 7
1.4.1 XOR problem and Parity problem 7
1.4.2 Sonar problem 10
1.5 Conclusions 10
1.6 Appendix 11
Chapter 2 Convergence of online gradient method for feedforward neural network with smoothing L1/2 regularization penalty 17
2.1 Introduction 17
2.2 Algorithm description 19
2.2.1 Online gradient method with L2 regularization 20
2.2.2 Online gradient method with L1/2 regularization 21
2.2.3 Online gradient method with smoothing L1/2 regularization 22
2.3 Convergence results 24
2.4 Numerical examples 24
2.4.1 Example 1: Parity problem 24
2.4.2 Example 2: function regression problem 26
2.5 Conclusions 28
2.6 Appendix 28
Chapter 3 Deterministic convergence analysis via smoothing group lasso regularization and adaptive momentum for Sigma-Pi-Sigma neural network 41
3.1 Introduction 41
3.2 Network structure and learning algorithm 44
3.2.1 Batch gradient algorithm with group lasso regularization and adaptive momentum 44
3.2.2 Batch gradient algorithm with smoothing group lasso regularization and adaptive momentum 47
3.3 Convergence results 48
3.4 Simulation results 49
3.4.1 Function approximation problems 49
3.4.2 K-dimensional Parity problem 52
3.4.3 Classification problems 54
3.5 Conclusions 56
3.6 Appendix 56
Chapter 4 Convergence analysis for Sigma-Pi-Sigma neural network based on some relaxed conditions 70
4.1 Introduction 70
4.2 Sigma-Pi-Sigma neural network and batch gradient algorithm 73
4.2.1 Sigma-Pi-Sigma neural network 73
4.2.2 Batch gradient algorithm for Sigma-Pi-Sigma neural network with regularization 74
4.3 Main results 75
4.4 Simulation results 77
4.4.1 Function approximation problems 77
4.4.2 Parity problems 79
4.4.3 Classification problems 79
4.5 Conclusions 82
4.6 Appendix 82
Chapter 5 Recurrent neural networks with smoothing regularization for regression and multiclass classification problems 95
5.1 Introduction 95
5.2 The model structure of RNN97
5.3 Gradient learning method in RNN with smoothing L1/2 regularization 100
5.4 Numerical simulations 103
5.4.1 XOR problems 103
5.4.2 Function approximation problems 104
5.4.3 Classification problems 105
5.5 Conclusions 107
Chapter 6 Weak and strong convergence analysis of Elman neural network via weight decay regularization.108
6.1 Introduction 108
6.2 Algorithm description 110
6.2.1 Batch gradient algorithm for Elman neural network 112
6.2.2 Batch gradient algorithm for Elman neural network with L2
regularization.113
6.2.3 Introduction of related definitions 114
6.3 Convergence results 115
6.4 Numerical examples.116
6.4.1 Function approximation 117
6.4.2 Real-word classification 118
6.4.3 XOR with two-cycle delay problem 120
6.5 Conclusions 122
6.6 Appendix 122
Chapter 7 Convergence of a gradient-based learning algorithm with penalty for ridge polynomial neural network 131
7.1 Introduction 131
7.2 Network structure description 133
7.2.1 Pi-Sigma neural network 133
7.2.2 Ridge polynomial neural network 134
7.3 Batch gradient learning algorithm for RPNN135
7.3.1 The original group lasso regularization algorithm 135
7.3.2 The smoothing group lasso regularization algorithm 137
7.4 Theorems of monotonicity and convergence 138
7.5 Numerical examples 139
7.5.1 Example 1: function approximation problem 139
7.5.2 Example 2: Parity problem 141
7.6 Conclusions 143
7.7 Appendix 143
Chapter 8 Regression and multiclass classification using sparse extreme learning machine via smoothing group L1/2 regularizer 152
8.1 Introduction 152
8.2 The preliminary ELM.154
8.3 Algorithm description 156
8.3.1 Group L1/2 ELM for hidden nodes and output weights 156
8.3.2 Smoothing group L1/2 ELM for hidden nodes and output weights 157
8.4 Experimental results 159
8.4.1 Experimental setup 159
8.4.2 Benchmarking with regression problems 161
8.4.3 Benchmarking with classification problems 166
8.5 Conclusions 168
Chapter 9 Smooth L0 regularization for extreme learning machine 170
9.1 Introduction 170
9.2 Extreme learning machine 172
9.3 Extreme learning machine with L0 regularization 174
9.4 Description of sparsity 175
9.5 Simulation results 177
9.5.1 Function regression problems 177
9.5.2 Real-word classification problems 181
9.6 Conclusions 183
Chapter 10 A hybrid model of extreme learning machine based on bat a