Ficha de revisão: Fundamentals of Image Classification and Neural Networks

📋 Course Outline

1. Image classification basics
2. Machine learning paradigms
3. Support vector machines
4. Neural network fundamentals
5. Bag of visual words
6. Neural network training
7. Forward propagation
8. Backpropagation algorithm
9. Activation functions
10. Image classification datasets
11. Transfer learning
12. Neural network loss functions

📖 1. Image classification basics

🔑 Key Concepts & Definitions

• Class of the object: The category or label assigned to an object within an image, such as "cat" or "dog," used for categorization in image classification tasks.

• Class label: A discrete identifier (e.g., "Car", "Tree") assigned to an object in an image, representing its category or class.

• Class scores estimation: The process of predicting numerical scores for each class, reflecting the likelihood or confidence that the object belongs to each class, often used to derive the final class label.

• Global features: Descriptive attributes extracted from the entire image, such as HOGs, LBPs, or Haar wavelets, capturing overall appearance or texture information for classification.

• Local features: Descriptors derived from specific regions or interest points in the image, such as SIFT + BoVW or SURF + BoVW, capturing local patterns and details relevant for distinguishing objects.

• Image classification as class scores estimation: The approach where the model outputs a set of class scores (or probabilities), which are then interpreted to assign a class label to the image.

📝 Essential Points

• Image classification involves identifying the class of the object shown in an image, often based on class scores estimation, which provides confidence levels for each class (see class scores estimation).

• Features used for classification can be global features (e.g., HOGs, LBPs, Haar wavelets) that describe the entire image, or local features (e.g., SIFT + BoVW, SURF + BoVW) that focus on interest points or regions.

• The bag of visual words (BoVW) model extracts local features, clusters them into codewords, and represents images as histograms of these codewords, which serve as feature descriptors for classifiers.

• Classifiers such as linear classifiers, SVMs, ensembles, and neural networks are employed to map features to class labels, often using class scores as intermediate outputs.

💡 Key Takeaway

Image classification relies on extracting global or local features to estimate class scores, which are then used to determine the object’s class label through various classifiers, enabling automated recognition of objects in images.

📖 2. Machine learning paradigms

🔑 Key Concepts & Definitions

Supervised learning: A machine learning paradigm where models are trained on labeled data, meaning each input has an associated ground truth output or label. The goal is to learn a mapping from inputs to outputs, minimizing the difference between predicted and true labels. (see source content for context)

Unsupervised learning: A paradigm where models are trained on unlabeled data, aiming to discover inherent structures or patterns within the data, such as clusters or associations, without explicit labels guiding the learning process. (see source content for context)

Semi-supervised learning: Combines aspects of supervised and unsupervised learning by training on a dataset that contains both labeled and unlabeled data. It leverages the limited labeled data to guide the learning process while exploiting the unlabeled data to improve model performance. (see source content for context)

📝 Essential Points

• Supervised learning is fundamental for tasks like image classification, where class labels (e.g., "Cat", "Dog") are used to train models such as neural networks, SVMs, and decision trees. It relies heavily on labeled datasets for effective training.
• Unsupervised learning is often used for feature extraction, data clustering, and anomaly detection, especially when labels are unavailable or costly to obtain.
• Semi-supervised learning is particularly useful when labeled data is scarce or expensive, allowing models to improve by utilizing large amounts of unlabeled data alongside limited labels.
• These paradigms form the basis for various algorithms and architectures, influencing how models are trained and evaluated in tasks like object detection and image recognition.

💡 Key Takeaway

Machine learning paradigms differ primarily in their use of labeled data: supervised learning relies on labels for direct mapping, unsupervised learning discovers patterns without labels, and semi-supervised learning balances both approaches to optimize learning efficiency.

📖 3. Support vector machines

🔑 Key Concepts & Definitions

• Support Vector Machines (SVMs): A supervised learning model introduced by Vladimir Vapnik (1995), designed for classification tasks by finding the optimal hyperplane that separates classes with the maximum margin.

• Hard Margin SVM: A variant of SVM that seeks a hyperplane separating classes without misclassification, assuming data is linearly separable. It maximizes the margin between the closest points of each class, called support vectors.

• Soft Margin SVM: An extension introduced by Vapnik (1995) to handle non-linearly separable data by allowing some misclassifications. It introduces slack variables to balance margin maximization and classification errors, making the model more robust to noise.

• Hinge Loss Function: A convex loss function used in SVMs, defined as $\max(0, 1 - y_i (w \cdot x_i + b))$, where $y_i$ is the true label. It penalizes points within the margin or misclassified, encouraging the model to maximize the margin.

• Multi-class SVMs: An extension of binary SVMs to handle multiple classes, often implemented via strategies like one-vs-rest or one-vs-one, to classify data into more than two categories.

📝 Essential Points

• SVMs aim to find the hyperplane that maximizes the margin, which is the distance between the hyperplane and the nearest data points (support vectors). This maximization leads to better generalization performance.

• The hard margin SVM works only when data is perfectly separable; otherwise, it cannot find a feasible solution. The soft margin SVM introduces slack variables $\xi_i$ to allow some points to violate the margin constraints, controlled by a regularization parameter $C$.

• The hinge loss function is central to SVM optimization, as it directly penalizes points that are within the margin or misclassified, guiding the model to improve the margin.

• Multi-class classification with SVMs is typically achieved by combining multiple binary classifiers, using methods like one-vs-rest, where a classifier is trained for each class against all others.

💡 Key Takeaway

Support vector machines are powerful classifiers that maximize the margin between classes, with the soft margin variant providing robustness to noise and non-separable data, primarily optimized through hinge loss. Multi-class SVMs extend this framework to handle multiple categories effectively.

📖 4. Neural network fundamentals

🔑 Key Concepts & Definitions

• Biological neuron: A nerve cell in the brain that receives inputs via dendrites, processes these signals, and transmits an output through its axon. Inputs can excite or inhibit the neuron, and the output is bounded within a finite range, with neurons interconnected to form complex networks.

• Artificial neuron model: A computational unit inspired by biological neurons, consisting of a linear combination of inputs (weighted sum plus bias) followed by a non-linear activation function, used to mimic biological neural processing in machine learning.

• Perceptron as a linear classifier: Developed by Rosenblatt (1957), the perceptron is a simple neural network that classifies data by applying a linear decision boundary, using the Heaviside step function as the activation to produce a discrete output.

• Multi-layer perceptron (MLP) architecture: A neural network composed of multiple layers of neurons (input, hidden, output), where each layer is fully connected to the next, enabling the modeling of complex, non-linear relationships in data.

• Fully-connected layers in neural networks: Layers where each neuron in one layer is connected to every neuron in the previous layer, allowing comprehensive information flow and feature combination across the network.

📖 5. Bag of visual words

🔑 Key Concepts & Definitions

• Bag of visual words (BoVW) model: A method that extracts relevant features (visual words) from images to build a dictionary of codewords, representing images as histograms of codeword counts, used for classification (source: Giacomo Tarroni).

• Feature extraction using interest points and descriptors: Techniques like SIFT or SURF detect interest points in images and describe them with feature vectors, capturing local visual information (source: Giacomo Tarroni).

• Clustering algorithm (e.g., K-means): A method that groups feature descriptors into clusters, where each cluster center becomes a codeword in the codebook, representing common visual patterns across images (source: Giacomo Tarroni).

• Histogram representation of images: A vector that counts the frequency of each codeword in an image, serving as a feature descriptor for classification tasks (source: Giacomo Tarroni).

• Using histograms as feature descriptors: The process of employing the histogram of codewords to train classifiers such as SVMs, enabling image categorization based on local features (source: Giacomo Tarroni).

📝 Essential Points

• The BoVW model begins with interest point detection in training images, followed by feature description using methods like SIFT or SURF. These descriptors are then clustered via algorithms like K-means to form a codebook of representative visual words (source: Giacomo Tarroni).

• Each image is represented by a histogram of codeword counts, which summarizes the distribution of visual patterns within the image. These histograms serve as feature vectors for training classifiers such as SVMs or neural networks (source: Giacomo Tarroni).

• During testing, the same process is repeated: interest points are detected, descriptors are computed, and histograms are generated using the pre-defined codebook. The resulting histograms are classified to determine the image's category (source: Giacomo Tarroni).

• The BoVW approach enables the use of local feature descriptors for image classification, bridging the gap between local pattern detection and global image understanding (source: Giacomo Tarroni).

💡 Key Takeaway

The Bag of Visual Words model transforms local image features into a histogram-based representation, allowing effective classification by capturing the distribution of visual patterns across images.

📖 6. Neural network training

🔑 Key Concepts & Definitions

• Training set with paired data and ground truth labels: A collection of input-output pairs used to train neural networks, where each input (feature vector) is associated with a known correct output (label) (see source content).
• Loss function for a single sample: A mathematical function that quantifies the discrepancy between the network's predicted output and the true label for one data point, such as $J_m(W, b) = \frac{1}{2} \| a^{(m)}(W, b) - y^{(m)} \|^2$.
• Overall training set loss function: The average of individual loss functions over all training samples, used as an objective to optimize: $J(W, b) = \frac{1}{M} \sum_{m=1}^M J_m(W, b)$.
• Minimizing loss function to find network parameters: The process of adjusting weights $W$ and biases $b$ via algorithms like gradient descent to reduce the loss, thereby improving the network's predictions (see source content).
• Regularisation in loss function (L1 and L2 norms): Additional terms added to the loss to penalize complex models, with L1 encouraging sparsity and L2 discouraging large weights, aiding generalization and avoiding overfitting (see source content).

📝 Essential Points

• The training process involves defining a loss function for each sample and averaging it over the entire dataset to guide parameter updates (see source).
• The loss function's minimization is achieved through iterative algorithms like gradient descent, which require calculating gradients via backpropagation (see source).
• Regularisation terms, such as L1 and L2 norms, are incorporated into the loss function to penalize large weights, promoting better generalization and reducing overfitting (see source).
• L1-norm regularisation encourages sparsity, effectively performing feature selection, while L2-norm regularisation penalizes large weights more strongly, promoting diffuse weight distributions (see source).
• Proper regularisation and choice of hyperparameters (e.g., regularisation coefficient $\lambda$) are crucial for training neural networks that generalize well to unseen data (see source).

💡 Key Takeaway

Training neural networks involves minimizing a loss function over paired data, with regularisation techniques like L1 and L2 norms helping to improve model generalization and prevent overfitting.

📖 7. Forward propagation

🔑 Key Concepts & Definitions

• Forward propagation: The process by which a neural network computes its output by passing input data through successive layers, applying weights, biases, and activation functions (see source content). It transforms input features into predicted outputs.

• Notation for weights, biases, inputs, activations, and layers:

  • $\mathbf{W}^{(l)}$: weight matrix connecting layer $l$ to layer $l+1$.
  • $\mathbf{b}^{(l)}$: bias vector for layer $l+1$.
  • $\mathbf{a}^{(l)}$: activation vector of layer $l$.
  • $\mathbf{z}^{(l)}$: input to neurons in layer $l$, before activation.
  • $\mathbf{x}$: input features (see source content).

• Matrix formulation of forward propagation:

  • At each layer $l+1$, the input to neurons is calculated as:
    $\mathbf{z}^{(l+1)} = \mathbf{W}^{(l)} \mathbf{a}^{(l)} + \mathbf{b}^{(l)}$
  • The activation output is then obtained by applying the activation function element-wise:
    $\mathbf{a}^{(l+1)} = f(\mathbf{z}^{(l+1)})$
  • This process repeats layer by layer, from input to output.

• Calculation of neuron inputs and activations layer by layer:

  • For each layer $l$, compute:
    1. Input to neurons: $\mathbf{z}^{(l+1)} = \mathbf{W}^{(l)} \mathbf{a}^{(l)} + \mathbf{b}^{(l)}$
    2. Activation: $\mathbf{a}^{(l+1)} = f(\mathbf{z}^{(l+1)})$
  • The process continues until the final layer produces the network's output.

📝 Essential Points

• Forward propagation is the fundamental step in neural network inference, where the input data is systematically transformed through each layer's linear and non-linear operations (see source content).
• The notation $\mathbf{W}^{(l)}$, $\mathbf{b}^{(l)}$, $\mathbf{a}^{(l)}$, and $\mathbf{z}^{(l)}$ helps formalize the process, enabling matrix operations that are computationally efficient.
• The matrix formulation simplifies the calculation across multiple neurons and layers, allowing vectorized implementation.
• At each layer, the input $\mathbf{z}^{(l+1)}$ is obtained via a linear combination of previous layer activations, then passed through an activation function $f$, such as sigmoid or ReLU, to produce the current layer's activations.
• The process is repeated sequentially from the input layer to the output layer, producing the network's prediction.

💡 Key Takeaway

Forward propagation efficiently computes the neural network's output by passing data through layers using matrix operations, combining weights, biases, and activation functions to transform inputs into predictions.

📖 8. Backpropagation algorithm

🔑 Key Concepts & Definitions

• Propagation of error from output layer backwards: The process of transmitting the discrepancy between the predicted output and the true label from the final layer of the neural network back through the preceding layers to update weights and biases, as developed by Rumelhart, Hinton, and Williams (1986).
• Use of chain rule in differentiation: A mathematical principle that allows the calculation of derivatives of composite functions by multiplying the derivatives of each function in the chain, fundamental for computing gradients during backpropagation.
• Calculation of gradients of loss function with respect to weights and biases: The process of determining how small changes in weights and biases affect the loss, enabling gradient descent optimization to minimize error, based on the derivatives obtained via backpropagation.

📝 Essential Points

Backpropagation is an algorithm introduced by Rumelhart, Hinton, and Williams (1986) that efficiently computes the gradients of the loss function with respect to all network parameters (weights and biases). It propagates the error from the output layer backwards through the network, utilizing the chain rule in differentiation to decompose the derivatives of the composite functions involved in neural network computations. This process involves calculating the partial derivatives of the loss with respect to activations, inputs, weights, and biases at each layer, enabling the use of gradient descent to iteratively update parameters. The core idea is to leverage the chain rule to avoid redundant calculations, making training deep neural networks computationally feasible.

💡 Key Takeaway

Backpropagation systematically computes the gradients needed for neural network training by propagating errors backwards through the network layers using the chain rule, enabling efficient optimization of weights and biases via gradient descent.

📖 9. Activation functions

🔑 Key Concepts & Definitions

• Activation functions in neurons: Mathematical functions applied to a neuron's input to introduce non-linearity, enabling neural networks to learn complex patterns. They mimic biological neurons' firing behavior by determining whether a neuron activates based on its input.

• Sigmoid (logistic) function: A smooth, S-shaped activation function defined as $f(z) = \frac{1}{1 + e^{-z}}$. It maps any real-valued input to a range between 0 and 1, useful for probabilistic interpretation in binary classification.

• Derivative of sigmoid function: The rate of change of the sigmoid function, given by $f'(z) = f(z)(1 - f(z))$. This derivative is essential for backpropagation during neural network training.

• Biological inspiration of activation functions: Activation functions are inspired by biological neurons, which fire only when inputs exceed a certain threshold, similar to how functions like sigmoid produce outputs based on input magnitude.

• Vanishing gradient problem: A training issue where the gradients (derivatives) of activation functions like sigmoid become very small (approach zero) for large positive or negative inputs, hindering effective learning in deep networks.

📝 Essential Points

• Activation functions in neurons serve as the core non-linear component that allows neural networks to approximate complex functions (see Biological neuron). The sigmoid function, introduced as a biologically inspired model, is historically significant but suffers from the vanishing gradient problem, especially in deep networks (see Sigmoid (logistic) function). Its derivative, $f'(z) = f(z)(1 - f(z))$, becomes very small when $f(z)$ approaches 0 or 1, which impairs weight updates during backpropagation (see Derivative of sigmoid function). This issue leads to slow convergence or training failure, known as the vanishing gradient problem. Alternatives like ReLU and its variants have been developed to mitigate this problem by maintaining larger gradients for certain input ranges.

💡 Key Takeaway

Activation functions in neurons, especially the sigmoid, are crucial for introducing non-linearity but can cause training difficulties like the vanishing gradient problem; understanding their biological basis and limitations guides the development of more effective functions like ReLU.

📖 10. Image classification datasets

🔑 Key Concepts & Definitions

• MNIST dataset: A widely used dataset for handwritten digit recognition, consisting of approximately 70,000 images of size 28x28 pixels across 10 classes, introduced by Y. LeCun et al. (1998). It serves as a benchmark for evaluating image classification models.

• Training and evaluation datasets: Collections of labeled images used to train machine learning models and assess their performance, respectively. These datasets provide the necessary data for supervised learning tasks, enabling models to learn features and generalize to unseen data.

• Use of datasets for training and evaluation: The process involves training models on a labeled dataset to learn patterns and then evaluating their accuracy or error rate on separate test data to measure generalization ability. Data augmentation (see section 10) can artificially expand datasets to improve model robustness.

📝 Essential Points

• Image classification datasets like MNIST are fundamental for developing and benchmarking neural networks and other classifiers, providing standardized data for comparison.

• The training set is used to optimize model parameters through learning algorithms such as gradient descent, while the evaluation set measures the model's ability to generalize to new, unseen images.

• Data augmentation techniques, such as affine transformations, are employed to artificially increase dataset size, helping models avoid overfitting and improve accuracy, especially when training data is limited.

• Large datasets like ImageNet (not explicitly detailed here) are crucial for training deep neural networks, but smaller datasets like MNIST remain popular for initial experimentation and benchmarking.

💡 Key Takeaway

Standard image classification datasets are essential tools for training, evaluating, and benchmarking models, with data augmentation playing a vital role in enhancing model performance and robustness.

📖 11. Transfer learning

🔑 Key Concepts & Definitions

• Transfer learning: A machine learning technique where a neural network pre-trained on a large, related dataset is adapted to a new, often smaller, dataset by re-training some layers while keeping others fixed, thus leveraging previously learned features (source: Justin Johnson, Lecture 7, cs231n).

• Using pre-trained networks for new tasks: The process involves taking a neural network trained on a large dataset (e.g., ImageNet), and re-initializing or fine-tuning specific layers to perform a different but related task, reducing training time and data requirements (source: Justin Johnson, Lecture 7, cs231n).

• Benefits of transfer learning for image classification: It enables models to achieve higher accuracy with less data, mitigates overfitting on small datasets, and accelerates training by utilizing learned feature representations from large datasets, especially effective with deep CNNs (source: Justin Johnson, Lecture 7, cs231n).

📝 Essential Points

• Transfer learning is particularly effective in deep CNNs where models pre-trained on large datasets like ImageNet serve as a starting point for different image classification tasks (source: Justin Johnson, Lecture 7, cs231n).

• The process involves pre-training a model on a big dataset, then re-initializing and retraining only a subset of layers or fine-tuning the entire network with a lower learning rate, especially when the second dataset is small (source: Justin Johnson, Lecture 7, cs231n).

• The number of layers re-initialized depends on the size and similarity of the second dataset; more layers can be re-trained if the second dataset is large, otherwise, layers are frozen to preserve learned features (source: Justin Johnson, Lecture 7, cs231n).

💡 Key Takeaway

Transfer learning allows neural networks to leverage knowledge from large, related datasets, making it a powerful strategy to improve performance and efficiency in image classification tasks, especially when data is limited.

📖 12. Neural network loss functions

🔑 Key Concepts & Definitions

• Loss function (see source content): A mathematical function that quantifies the discrepancy between the neural network's predicted output and the ground truth labels, guiding the optimization process during training.

• Mean squared error (MSE) loss: A common loss function for regression tasks, defined as the average of the squared differences between predicted values and true values, i.e., $J = \frac{1}{M} \sum_{i=1}^M (a_i - y_i)^2$. It penalizes larger errors more heavily and encourages the network to produce predictions close to the actual values.

• Cross-entropy loss: A loss function used primarily for classification tasks, measuring the difference between the true probability distribution $p$ and the estimated distribution $q$. For binary classification, it is expressed as $J = -[ y \log(f(z)) + (1 - y) \log(1 - f(z)) ]$, where $f(z)$ is the network's output after sigmoid activation. It effectively penalizes incorrect probability estimates.

📝 Essential Points

• Loss functions are essential in training neural networks as they provide a scalar measure of prediction accuracy, which is minimized during optimization (see loss function). They directly influence how the network's parameters are updated via algorithms like gradient descent.

• For regression tasks, mean squared error loss is typically used because it penalizes deviations quadratically, promoting predictions that are close to the true continuous values.

• For classification, especially binary classification, cross-entropy loss is preferred because it aligns with the probabilistic interpretation of the network's output (via sigmoid) and penalizes incorrect class probability estimations effectively.

• In multi-class classification, the softmax function combined with the cross-entropy loss is employed to produce a probability distribution over classes, with the loss measuring the divergence between predicted and true class distributions.

• The role of loss functions extends beyond measuring error; they also serve as the objective function that guides the backpropagation process, enabling the calculation of gradients necessary for updating network weights.

💡 Key Takeaway

Loss functions are fundamental to neural network training, translating prediction errors into a scalar value that guides parameter updates; mean squared error is suited for regression, while cross-entropy is optimal for classification tasks.

📊 Synthesis Tables

Aspect	Support Vector Machines (Vladimir Vapnik)	Neural Networks (Rosenblatt, 1957)
Core Concept	Finds the hyperplane maximizing margin between classes	Mimics biological neurons with weighted inputs and activation functions
Margin	Maximizes the distance to support vectors	Not explicitly focused on margin; learns decision boundaries via weights
Handling Non-separable Data	Soft margin with slack variables	Capable of modeling complex, non-linear decision boundaries with multiple layers
Loss Function	Hinge loss	Typically uses loss functions like mean squared error or cross-entropy
Multi-class Extension	One-vs-rest, one-vs-one strategies	Multi-layer perceptrons with output layers for multiple classes

⚠️ Common Pitfalls & Confusions

1. Confusing hard margin SVM with soft margin SVM; the latter is more practical for real-world noisy data.
2. Misunderstanding the role of support vectors; only support vectors influence the decision boundary.
3. Overlooking the importance of kernel functions in SVMs for non-linear classification.
4. Assuming neural networks always require deep architectures; shallow networks can suffice for simple tasks.
5. Confusing the perceptron’s linear decision boundary with the complex boundaries learned by multi-layer neural networks.
6. Ignoring the need for activation functions (e.g., ReLU, sigmoid) to introduce non-linearity in neural networks.
7. Misinterpreting the purpose of the bias term in neurons as just an offset, rather than a learnable parameter influencing the decision boundary.

✅ Exam Checklist

• Know Vapnik's concept of the maximum margin in SVMs and the distinction between hard and soft margin variants.
• Understand the hinge loss function and its role in SVM optimization.
• Be able to explain the biological inspiration behind neural networks, including the structure of a biological neuron.
• Recall Rosenblatt's perceptron as a simple linear classifier and its limitations.
• Describe the architecture of a multi-layer perceptron (MLP) and its capacity to model complex decision boundaries.
• Understand the importance of activation functions such as sigmoid, ReLU, and tanh in neural networks.
• Know the difference between global and local features in image classification.
• Be familiar with the machine learning paradigms: supervised, unsupervised, semi-supervised.
• Recognize the purpose and process of feature extraction in Bag of Visual Words models.
• Understand transfer learning and its application in image classification tasks.
• Know the common loss functions used in neural network training, such as cross-entropy and mean squared error.
• Be able to compare SVMs and neural networks in terms of their strengths, limitations, and typical use cases.