总体最小二乘问题的有效在线学习算法

This book reports the developments of the Total Least Square (TLS) algorithms for parameter estimation and adaptive filtering.Specifically,it introduces the authors' latest achievements in the past 20 years,including the recursive TLS algorithms,the approximate inverse power iteration TLS algorithm,the neural based MCA algorithm,the neural based SVD algorithm,the neural based TLS algorithm,the TLS algorithms under non-Gaussian noises,performance analysis methods of TLS algorithms,etc.In order to faster the understanding and mastering of the new methods provided in this book for readers,before presenting each new method in each chapter,a specialized section is provided to review the closely related several basis models.Throughout the book,large of procedure of new methods are provided,and all new algorithms or methods proposed by us are tested and verified by numerical simulations or actual engineering applications.Readers will find illustrative demonstration examples on a range of industrial processes to study.Readers will find out the present deficiency and recent developments of the TLS parameter estimation fields,and learn from the authors' latest achievements or new methods around the practical industrial needs.In my opinion,this book can be assimilated by advanced undergraduates and graduate students,as well as statisticians,because of the new tools in data analysis,applied mathematics experts,because of the novel theories and techniques that we propose,engineers,above all for the applications in control,system identification,computer vision,and signal processing.

1 Introduction 1

1.1 An Overview of Total Least Square 1

1.1.1 Total Least Square Problems 1

1.1.2 Total Least Square Methods 3

1.1.3 Related Monographs for Total Least Square 4

1.2 Aim and Main Features of This Book 6

1.2.1 Aim of This Book 6

1.2.2 Main Features of This Book 6

1.3 Organization of This Book 8

References 9

2 Total Least Square Problems 13

2.1 Preliminaries 13

2.1.1 Overdetermined Linear Equations and SVD of Matrix 13

2.1.2 Ordinary Least Squares (OLS) Problems 16

2.2 Classical TLS Problem 16

2.2.1 Basic TLS Problems 17

2.2.2 OLS and TLS Geometric Considerations 18

2.2.3 Multidimensional TLS Problem 20

2.2.4 Non-generic Unidimensional TLS Problem 23

2.3 Extension of TLS Problem 25

2.3.1 Mixed OLS-TLS Problem 25

2.3.2 Statistical Properties and Validity 26

2.3.3 Basic Data Least Squares Problem 27

2.3.4 Generalized TLS Problem 28

2.3.5 Weighted TLS Problem 31

2.3.6 Constrained TLS Problem 32

2.3.7 Structured TLS Problem 33

2.3.8 TLS in Nonlinear EIV Models 34

2.3.9 TLS Under Non-gaussian Noises 35

2.4 Summary 36

References 36

3 Total Least Square Methods 39

3.1 TLS Solution 39

3.2 Partial TLS Algorithm 42

3.3 Iterative Computation Methods 42

3.3.1 Direct Versus Iterative Computation Methods 43

3.3.2 Inverse Iteration 43

3.3.3 Chebyshev Iteration 44

3.3.4 Lanczos Methods 44

3.3.5 Rayleigh Quotient Iteration 45

3.4 Rayleigh Quotient Minimization Non-neural Methods 45

3.4.1 Davila's Recursive TLS Algorithm 46

3.4.2 Feng's Fast Recursive TLS Algorithm 46

3.4.3 Feng's Fast Approximate Inverse Power Iterative TLS Algorithm 47

3.5 Neural Networks Methods for TLS 47

3.5.1 Neural Networks Methods for MCA 47

3.5.2 Neural Networks Methods for SVD 49

3.5.3 Neural Networks that Iterate Only in the TLS Hyperplane 50

3.6 Other Methods of Extended TLS Problems 52

3.6.1 Weighted TLS Algorithms 52

3.6.2 Constrained TLS Algorithm 53

3.6.3 Structured TLS Solution 53

3.6.4 TLS in Nonlinear EIV Models 54

3.6.5 TLS Under Non-Gaussian Noises 54

3.7 Conclusions 55

References 55

4 Fast Recursive TLS Algorithms 59

4.1 Introduction 59

4.2 Review of Davila's RTLS Algorithm for FIR Adaptive Filtering 61

4.2.1 RLS Filter Bias and Mean Squared Error for IR Estimation 61

4.2.2 TLS for Adaptive FIR Filters 63

4.2.3 Recursive TLS Algorithm 64

4.3 A Fast Recursive TLS Algorithm for Adaptive FIR Filtering 66

4.3.1 TLS Problems in Signal Processing 67

4.3.2 Landscape of Criterion 69

4.3.3 New RTLS Algorithm 70

4.3.4 Convergence Analysis 72

4.3.5 Simulations and Conclusions 73

4.4 A Fast Recursive TLS Algorithm for Adaptive IIR Filtering 75

4.4.1 TLS Problems in Adaptive IIR Filtering and N-RTLS Algorithm 75

4.4.2 Algorithm Convergence 80

4.4.3 Simulations and Conclusions 82

4.5 Summary 85

References 86

5 Approximate Inverse Power Iteration TLS Algorithm 89

5.1 Introduction 89

5.2 Review of Inverse Power Iteration 90

5.3 AIP Iteration Algorithm for Adaptive TLS FIR Filtering 92

5.3.1 Preliminaries 92

5.3.2 AIP Iteration Algorithm 95

5.3.3 Algorithm Convergence 99

5.3.4 Simulations Examples and Conclusion 102

5.4 An AIP Algorithm for Adaptive Extraction of Minor Subspace 104

5.4.1 Proposed Adaptive Tracking Algorithm of MS 104

5.4.2 Theoretical Analysis of Algorithm 107

5.5 Summary 112

References 113

6 Neural-Based MCA Algorithms for Adaptive TLS 115

6.1 Introduction 115

6.2 Review of Neural-Network-Based MCA Algorithms 117

6.2.1 Oja's MCA Algorithms 117

6.2.2 Self-stabilizing MCA Algorithms 118

6.2.3 Orthogonal Oja's Algorithms 118

6.2.4 Coupled MCA Algorithms 118

6.3 Novel Parallel Multiple MCs Extraction Algorithm by Diagonal Matrix 119

6.3.1 Preliminary and Novel Algorithm 119

6.3.2 Fixed Points Analysis 121

6.3.3 Stability on the Manifold 124

6.3.4 Simulation Experiments and Conclusion 126

6.4 A Weighted Information Criterion and Multiple MC Algorithm 128

6.4.1 Preliminaries 128

6.4.2 A Weighted Information Criterion and Its Landscape 129

6.4.3 Adaptive Multiple MCs Extraction Algorithms 134

6.4.4 Convergence Analysis 137

6.4.5 Simulations Experiments and Conclusions 139

6.5 A Coupled Minor Component Analysis Algorithm 147

6.5.1 Coupled Dynamical System 147

6.5.2 Coupled MCA Learning Algorithms 149

6.5.3 Analysis of Convergence and Self-stabilizing Property 151

6.5.4 Simulation and Conclusions 153

6.6 Summary 158

References 158

7 Neural-Based SVD Algorithms for Adaptive TLS 161

7.1 Introduction 161

7.2 Review of Neural-Based SVD Algorithms 163

7.2.1 Parallel Learning Algorithms on Double Stiefel Manifold 163

7.2.2 Cross-Associative Neural Network for SVD (CANN) 165

7.2.3 Coupled SVD of a Cross-Covariance Matrix 167

7.3 A Neural Network for SVD of Non-squared Data Matrix 169

7.3.1 A Novel Recurrent Neural Network 169

7.3.2 Stability Analysis of Algorithm 172

7.3.3 Simulation Experiments and Conclusions 174

7.4 A Fast and Effective Neural Network Algorithm to Perform SVD 176

7.4.1 Preliminary 177

7.4.2 Novel Information Criterion and Algorithm 178

7.4.3 Convergence Analysis 180

7.4.4 Experiments and Conclusions 184

7.5 Coupled Neural Network Algorithm for PST Extraction 192

7.5.1 Novel Information Criterion and Its Coupled System 193

7.5.2 Experiments and Conclusions 196

7.6 Summary 198

References 199

8 Neural-Based TLS Algorithms 201

8.1 Introduction 201

8.2 Review of Neural-Based TLS Algorithms 202

8.2.1 The Hopfield-Like Neural Network of Luo,Li,and He 202

8.2.2 The Linear Neuron of Gao,Ahmad,and Swamy 203

8.2.3 The Linear Neurons of Cichocki and Unbehauen 205

8.2.4 The TLS EXIN Neural Network and Its Modified Version 206

8.2.5 The LS-TLS Neural Algorithm of Bruce and Williamson 206

8.3 A Self-stabilizing Neural Algorithm for TLS Filtering 209

8.3.1 The TLS Linear Neuron with a Self-stabilizing Algorithm 209

8.3.2 Self-stabilizing and Stability 211

8.3.3 Simulation of Algorithm 216

8.4 Conclusion 221

References 221

9 TLS Algorithm Under Non-Gaussian Noises 223

9.1 Introduction 223

9.2 Review of the LS/TLS Method Under Non-Gaussian Noises 224

9.2.1 The LS Method Under Non-Gaussian Noise 224

9.2.2 TLS Algorithm with Input and Output of the Same Noise Intensity 228

9.3 Mixed LS-TLS Algorithm Under Non-Gaussian Noises 232

9.3.1 Partial Input and Output Contaminated with Non-Gaussian Noises 232

9.3.2 Disparate Input and Output Non-Gaussian Noises 235

9.3.3 Input Non-Gaussian Noises Only 238

9.4 Simulation Experiments and Analysis 241

9.4.1 Experiment of NG-TLS Algorithm 241

9.4.2 Experiment of NG-LS-TLS Algorithm 243

9.5 Summary 246

References 246

10 Performance Analysis Methods of TLS Algorithms 247

10.1 Introduction 247

10.2 Review of the Analysis Methods of MCA/TLS Algorithms 248

10.2.1 Deterministic Continuous-Time System Method 248

10.2.2 Stochastic Discrete-Time System Method 249

10.2.3 Lyapunov Function Approach 250

10.2.4 Fixed Pointed Method 251

10.2.5 Deterministic Discrete-Time System Method 252

10.3 Convergence of Novel Generalized MCA Algorithm via DDT 252

10.3.1 Algorithm Presentation 252

10.3.2 Self-stabilizing Analysis 254

10.3.3 Dynamic Characteristic of the GChen Algorithm 255

10.3.4 Numerical Simulations and Conclusions 263

10.4 Summary 269

References 269