Motivation and preliminariesrisk minimization erm principle

12 1 Motivation and Preliminaries

Risk Minimization (ERM) principle, e.g. BP, adjust the model parameters, such that the resulting mapping function, f : IRD−→ IRC, fits the training data. On the other hand, the Structural Risk Minimization (SRM), e.g. SVMs, attempts to find the models with low Vapnik-Chervonenkis (VC) dimension [169]. This is a core concept, which relates to the interplay between how complex the model is and the capacity of generalization it can achieve. Either way, the objective consists of exploiting the observed data to build models that can make predictions about the output values of unseen input vectors [18].

(e.g. in a neural network model this value resorts to the odds that the sample belongs to class i).

In the case of unsupervised learning, typically the goal of the algorithms consists of producing a set of J informative features, h = [h1,h2,...,hJ] ∈ IRJ, for each input vector, x ∈ IRD. By analogy, the extracted features’ vectors, {h1,h2,...,hN}, form a feature matrix, H ∈ IRN×J, where each row contains a feature vector hi ∈ IRJ. Eventually, the extracted features can compose a basis for creating better supervised models. This process is illustrated in Figure 1.4.