ML – Core Concepts

Introduction

Machine learning (ML) encompasses a family of methods that learn patterns from data in order to make predictions, classify observations, or uncover structure. This document provides a concise foundation for understanding core ML concepts that apply across many algorithms, including variable types, encoding, scaling, distance functions, and model evaluation. These concepts form the basis for later algorithm-specific notes such as k-Nearest Neighbors (k-NN), regression models, and decision trees.

ML’s Position Within the AI Hierarchy:

        Artificial Intelligence (broad goal: intelligent behavior)
        |
        |– Symbolic AI (logic, rules, planning)
        |
        |– Machine Learning (learning patterns from data)
        |     |– Supervised learning
        |     |– Unsupervised learning
        |     |– Reinforcement learning
        |     |– Generative modeling
        |     \– Probabilistic inference (Naive Bayes, etc.)
        |
        \– Other subfields (NLP, robotics, etc.)

Learning Paradigms

Machine learning methods are traditionally grouped into a small number of categories based on the presence or absence of target labels. While these categories are widely used, the terminology is not always logically consistent. This section provides both the standard definitions and a brief discussion of their conceptual limitations.

Supervised Learning

In supervised learning, each observation contains predictor variables  and a known target variable . The model learns a mapping

by comparing its predictions to the known outcomes. This comparison produces an error signal that guides the learning process.

A Note on Terminology

Although the term supervised learning suggests that the model is actively supervised during training, the field defines supervision solely by the presence of labeled data, not by whether the model actually uses those labels to adjust internal parameters.

Under this stricter definition, some methods commonly labeled as supervised would logically fall into hybrid or alternative categories.

Unsupervised Learning

Semi-Supervised Learning

Semi-supervised learning uses a combination of labeled and unlabeled data. A small labeled subset provides guidance, while the larger unlabeled portion helps capture structure in the feature space.

Self-Supervised Learning

Although no human-provided labels exist, the task is framed as supervised because the model learns from a constructed target.

Weak Supervision

Weak supervision uses noisy, heuristic, or programmatically generated labels. These labels may be: incomplete, imprecise, or inconsistent.

The goal is to leverage imperfect supervision at scale.

Transfer Learning

Transfer learning uses knowledge gained from one task to improve performance on a different, often related task. Labels may exist for the source task but not for the target task.

Reinforcement Learning

Reinforcement learning (RL) differs fundamentally from supervised learning. Instead of labeled examples, the model receives rewards or penalties based on its actions. The objective is to learn a policy that maximizes cumulative reward.

Predictor and Target Variables

Predictor Variables

Predictor variables (features) are the inputs used to make predictions. For an observation , the predictor vector is:

Predictors may be continuous, binary, nominal, or ordinal. Their representation directly affects model behavior, especially in algorithms that rely on distance or similarity.

Target Variable

Supervised learning requires labeled pairs  to train a model.

Variable Types

Machine learning models operate on predictor variables that may take different forms. The type of each variable determines how it must be represented, encoded, and scaled before modeling. The main categories encountered in practice are:

• Continuous variables
  Real-valued numeric predictors (e.g., income, temperature). These typically require normalization or standardization so that variables on larger scales do not dominate model behavior.
• Binary variables
  Two-level categorical variables encoded as 0/1 (e.g., yes/no, male/female). These integrate naturally into most ML algorithms.
• Nominal categorical variables
  Categories with no inherent ordering (e.g., state, color). These must be converted to dummy (one-hot) encoded variables so that models can interpret them numerically.
• Ordinal categorical variables
  Categories with a meaningful order (e.g., low < medium < high). These may be encoded as integers if the ordering is meaningful, or dummy-encoded if the numeric spacing between levels is not interpretable.

Proper handling of variable types is essential because many ML algorithms rely on numeric representations, and some (e.g., distance-based methods) are highly sensitive to scale and encoding choices.

Encoding of Categorical Variables

Categorical predictors must be transformed into numeric form before modeling. Common encoding strategies include:

• Dummy (One-Hot) Encoding
  Creates a binary indicator for each category. Suitable for nominal variables with no inherent order.
• Ordinal Encoding
  Assigns integer values to ordered categories. Appropriate when the ordering is meaningful and the spacing between levels is interpretable.
• Binary Encoding / Hashing (Advanced)
  Used for high-cardinality categorical variables to reduce dimensionality.

The choice of encoding affects model interpretability, dimensionality, and how similarity or distance is computed.

Normalization and Scaling

Many ML algorithms are sensitive to the scale of predictor variables. Features measured on larger numeric ranges can dominate model behavior unless standardized. Common scaling methods include:

• Standardization (Z-Score Scaling)

• Centers each variable at zero mean with unit variance. Essential for distance-based methods and algorithms that assume normally distributed predictors.
• Min–Max Scaling

• Rescales variables to the interval . Useful when preserving the shape of the original distribution is important.
• Robust Scaling
  Uses median and interquartile range (IQR), making it less sensitive to outliers.

Scaling ensures that all predictors contribute appropriately to model behavior and prevents variables with large numeric ranges from dominating distance or gradient calculations.

Distance-Based Similarity

Many machine learning methods rely on a notion of similarity between observations. Similarity is often defined through a distance function

which quantifies how different two observations are in the feature space.

Distance-based reasoning appears in a wide range of ML algorithms, including:

• nearest-neighbor methods,
• clustering algorithms (e.g., k-means, hierarchical clustering),
• anomaly detection,
• recommender systems,
• dimensionality reduction and manifold learning.

The choice of distance metric determines the geometry of the feature space and directly influences which observations are considered similar.

Distance Metrics

Several distance functions are commonly used in machine learning. Each metric encodes different assumptions about the structure of the data.

Euclidean Distance

The most widely used metric for continuous numerical features:

• Sensitive to scale; features must be standardized.
• Represents straight-line (L2) distance in Euclidean space.

Manhattan Distance (L1)

Sum of absolute differences:

• More robust to outliers than Euclidean distance.
• Useful in high-dimensional or sparse spaces.

Minkowski Distance

Generalization of Euclidean and Manhattan distances:

•  gives Manhattan distance.
•  gives Euclidean distance.
• Allows tuning the geometry of the feature space.

Chebyshev Distance

Maximum coordinate difference:

• Useful when movement in any direction has equal cost.
• Appears in grid-based or chessboard-like problems.

Hamming Distance

Used for categorical or binary attributes. Counts the number of positions where values differ:

Binary case:

• Appropriate for discrete or encoded categorical variables.
• Measures mismatches rather than magnitude.

Cosine Distance

Measures orientation rather than magnitude:

• Common in text mining and high-dimensional sparse vectors.
• Ignores absolute scale; focuses on direction.

Mahalanobis Distance

Accounts for covariance structure of the data:

where  is the covariance matrix.

• Scales features by their variance and correlation.
• Useful when predictors have different variances or are correlated.

Basic Statistical Concepts

Several core statistical quantities appear throughout machine learning. These concepts describe how data vary, how features relate to one another, and how model errors are quantified.

Mean

For observations , the sample mean is:

Variance

Variance measures how far observations spread around the mean:

Standard Deviation

Standard deviation is the square root of variance:

It is expressed in the same units as the original variable, making it easier to interpret.

Covariance

Covariance measures how two variables vary together:

Correlation

Correlation standardizes covariance to the interval :

Correlation is scale-invariant and indicates the strength and direction of linear association.

These statistical quantities form the basis for understanding model error, feature relationships, and the behavior of many ML algorithms.

Naive Bayes Theorem and Its Importance

Naive Bayes is a probabilistic classification method based on Bayes’ Theorem with the simplifying assumption that features are conditionally independent given the class. Despite this assumption rarely holding exactly, the method is fast, robust, and highly effective in many real-world applications.

Bayes’ Theorem

Bayes’ Theorem provides a principled way to update the probability of a hypothesis after observing new evidence:

Here:

•  is the hypothesis (e.g., a class label),
•  is the observed evidence (features),
•  is the prior probability,
•  is the likelihood,
•  is the posterior probability.

This theorem forms the mathematical foundation of the Naive Bayes classifier.

The Naive Independence Assumption

Naive Bayes assumes that all features are conditionally independent given the class:

This assumption simplifies computation dramatically and allows the model to scale to high-dimensional data.

Naive Bayes Classifier

For each class , the classifier computes the posterior probability:

The predicted class is the one with the highest posterior probability.

Why Naive Bayes Is Important

Naive Bayes is foundational in machine learning for several reasons:

• Fast and scalable. Training reduces to counting frequencies or fitting simple distributions.
• Effective in high dimensions. Works well even when the number of features is large.
• Surprisingly robust. Performs well even when independence assumptions are violated.
• Backbone of text classification. Ideal for spam filtering, sentiment analysis, and document categorization.
• Teaches probabilistic reasoning. Provides clear intuition for priors, likelihoods, and posteriors.
• A generative model. Models the joint distribution , forming a bridge to more advanced probabilistic models.

One-Sentence Memory Trick

Naive Bayes = Bayes’ Theorem + independence assumption + fast, surprisingly strong classification.

Bias, Variance, and the Bias–Variance Tradeoff

Understanding model performance requires distinguishing between two fundamental sources of error: bias and variance. These concepts originate in basic statistics but play a central role in machine learning.

Statistical Variance and Standard Deviation

For a variable  with observations , the sample variance is:

and the standard deviation is:

Variance measures how spread out the values are around the mean. These definitions form the basis for understanding variance in machine learning models.

Bias in Machine Learning

Variance in Machine Learning

The Bias–Variance Tradeoff

Model performance reflects a balance between bias and variance. Increasing model flexibility typically:

• decreases bias,
• increases variance.

Conversely, restricting model flexibility:

• increases bias,
• decreases variance.

The optimal model minimizes total expected error by striking the right balance between these two forces.

Interpreting Bias and Variance in Practice

• High bias → underfitting, poor performance on both training and test data.
• High variance → overfitting, excellent training performance but poor generalization.
• Balanced models → moderate training error and strong test performance.

Cross-validation is commonly used to diagnose and manage the bias–variance tradeoff.

Classification and Regression

Supervised learning problems fall into two broad categories depending on the type of target variable.

Classification

Regression

Different evaluation metrics are used for classification and regression tasks.

Null Hypothesis and Error Types

Null Hypothesis (H₀)

The null hypothesis represents the default assumption: no effect, no difference, and no relationship. Statistical tests evaluate whether the observed data provide enough evidence to reject this assumption.

In classical hypothesis testing, we never “accept” the null hypothesis. Instead, we either:

• reject  (evidence suggests an effect), or
• fail to reject  (insufficient evidence to claim an effect).

This decision framework creates the possibility of Type I and Type II errors.

Type I Error (False Positive)

A Type I error occurs when we reject a true null hypothesis. In practical terms, we detect something that is not actually present.

Examples include:

• concluding a feature is predictive when it is not,
• flagging a legitimate transaction as fraud,
• diagnosing a healthy patient as sick.

This corresponds to the False Positive (FP) cell in a confusion matrix. The probability of a Type I error is the significance level, .

Type II Error (False Negative)

A Type II error occurs when we fail to reject a false null hypothesis. In practical terms, we miss something that is actually present.

Examples include:

• failing to detect a real relationship,
• allowing a fraudulent transaction to pass,
• missing a disease that is present.

This corresponds to the False Negative (FN) cell in a confusion matrix. The probability of avoiding a Type II error is the power of the test, .

Connection to Machine Learning

The logic of hypothesis testing maps directly onto binary classification:

Statistical Testing	Machine Learning
Reject	Predict positive
Fail to reject	Predict negative
Type I error	False Positive (FP)
Type II error	False Negative (FN)
Significance level ()	Precision tradeoff
Power ()	Recall tradeoff

One-Sentence Memory Trick

Type I = False Positive = detecting something that is not there.
Type II = False Negative = missing something that is there.

Confusion Matrix

For classification problems, performance is commonly summarized using a confusion matrix, which compares predicted vs. actual class labels.

Where:

• : true positives,
• : true negatives,
• : false positives,
• : false negatives.

These quantities form the basis for several classification metrics.

Classification Metrics

• Accuracy: proportion of correctly classified observations.

• Sensitivity (Recall, True Positive Rate):

• Specificity (True Negative Rate):

• Precision: proportion of predicted positives that are correct.

• F1 Score: harmonic mean of precision and recall.

• Cohen’s Kappa: agreement beyond chance.

• where  is observed accuracy and  is expected accuracy by chance.

Regression Metrics

For continuous targets, error-based measures quantify how close predictions are to true values.

• Mean Absolute Error (MAE):

• Root Mean Squared Error (RMSE):

• Coefficient of Determination ():

Interpreting Model Performance Metrics

Evaluation metrics quantify different aspects of model behavior. Understanding what each metric measures and what it does not measure is essential for interpreting results correctly.

Interpreting Classification Metrics

• Accuracy
  Measures overall correctness. Useful when classes are balanced, but misleading when one class dominates. A model predicting the majority class can achieve high accuracy without learning meaningful structure.
• Sensitivity (Recall)
  Measures the ability to detect positive cases. High sensitivity is critical when false negatives are costly (e.g., disease detection). A model with high sensitivity but low precision may over-predict positives.
• Specificity
  Measures the ability to correctly identify negatives. Important when false positives are costly (e.g., fraud alerts). Sensitivity and specificity often trade off.
• Precision
  Measures how reliable positive predictions are. High precision is important when acting on a positive prediction is expensive or risky.
• F1 Score
  Balances precision and recall. Useful when classes are imbalanced and neither precision nor recall alone is sufficient.
• Cohen’s Kappa
  Adjusts accuracy for agreement expected by chance. More informative than accuracy when class imbalance is present.

Interpreting Regression Metrics

• MAE
  Measures average error magnitude. Interpretable in the same units as the target. Robust to outliers.
• RMSE
  Penalizes large errors more heavily. Useful when large deviations are especially undesirable. More sensitive to outliers than MAE.
• 
  Measures proportion of variance explained. High  does not guarantee good predictions; a model can have high  but poor generalization. Negative values indicate performance worse than predicting the mean.

General Interpretation Principles

• No single metric is sufficient; multiple metrics provide a more complete picture.
• Metrics must be interpreted in the context of class balance, cost of errors, and domain requirements.
• High performance on training data does not imply generalization; cross-validation is essential.
• Threshold-dependent metrics (e.g., precision, recall) may change dramatically with different decision thresholds.

Choosing the Right Metric

Different problems emphasize different types of errors. The choice of metric should reflect the cost structure of the domain:

• Use sensitivity (recall) when missing a positive case is costly (e.g., medical screening).
• Use specificity when false alarms are costly (e.g., fraud alerts).
• Use precision when acting on a positive prediction is expensive or risky.
• Use F1 score when classes are imbalanced and both precision and recall matter.
• Use MAE when average error magnitude is the priority and robustness to outliers is desired.
• Use RMSE when large errors are especially undesirable.
• Use  to measure variance explained, not predictive accuracy.

Common Pitfalls in Model Evaluation

• Accuracy paradox: high accuracy can occur even when the model fails to detect minority classes.
• Imbalanced classes: metrics like precision, recall, and F1 are more informative than accuracy.
• Overfitting: strong performance on training data does not imply generalization; cross-validation is essential.
• Threshold dependence: precision and recall vary with the decision threshold; ROC and PR curves help visualize this.
• Data leakage: information from the test set must not influence training or preprocessing.

Overfitting and Underfitting

Model performance depends not only on the choice of algorithm and metrics, but also on how well the model balances complexity and generalization. Two common failure modes are overfitting and underfitting.

Underfitting

Underfitting occurs when a model is too simple to capture the underlying structure of the data. High bias leads to systematic errors.

Characteristics include:

• poor performance on both training and test data,
• overly smooth or simplistic predictions,
• failure to capture important relationships.

Examples:

• using a linear model for a nonlinear pattern,
• choosing a very large value of  in k-NN.

Overfitting

Overfitting occurs when a model is too flexible and captures noise or random fluctuations in the training data. High variance leads to unstable predictions.

Characteristics include:

• excellent performance on training data,
• poor generalization to new data,
• highly variable predictions across different samples.

Examples:

• using  in k-NN,
• deep decision trees without pruning,
• overly complex polynomial regression.

Diagnosing Overfitting and Underfitting

Evaluation metrics reveal characteristic patterns:

• Underfitting: high training error and high test error.
• Overfitting: low training error but high test error.

Cross-validation is a standard tool for detecting and mitigating both issues.

Relationship to the Bias–Variance Tradeoff

Underfitting corresponds to high bias and low variance. Overfitting corresponds to high variance and low bias.

Effective model development seeks a balance between these extremes to minimize total expected error.

Dimensionality and PCA

High-dimensional datasets create geometric and statistical challenges that make many machine learning algorithms unstable or ineffective. Principal Component Analysis (PCA) provides a principled way to reduce dimensionality while preserving the dominant structure in the data. This section summarizes the curse of dimensionality, the role of eigenvalues and eigenvectors, and the computation of principal components.

The Curse of Dimensionality

As the number of features  increases, several problems emerge:

• Distances lose meaning. In high dimensions, the nearest and farthest points become nearly the same distance apart.
• Data becomes sparse. The volume of the feature space grows exponentially, so fixed-size datasets occupy a tiny fraction of the space.
• Overfitting becomes easier. Models can memorize noise rather than learn structure.
• Visualization and intuition break down. Human reasoning does not extend beyond three dimensions.

These effects degrade the performance of distance-based algorithms and increase model variance.

Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors describe the intrinsic directions and magnitudes of variation in a linear transformation. For a square matrix , an eigenvector  and eigenvalue  satisfy:

• Eigenvectors represent principal directions.
• Eigenvalues represent the strength of variation along those directions.

These concepts form the mathematical foundation of PCA.

Computing Eigenvalues

Eigenvalues are obtained by solving the characteristic equation:

The roots of this polynomial are the eigenvalues of .

Computing Eigenvectors

For each eigenvalue , eigenvectors are found by solving the homogeneous system:

Any non-zero solution is an eigenvector associated with .

Principal Component Analysis (PCA)

PCA identifies the directions (principal components) along which the data vary the most. Given a standardized dataset, PCA proceeds as follows:

1. Compute the covariance matrix .
2. Compute eigenvalues and eigenvectors of .
3. Rank eigenvalues from largest to smallest.
4. Select the top  eigenvectors (principal components).
5. Project the data onto these components.

The resulting lower-dimensional representation captures the dominant structure in the data.

How PCA Mitigates High-Dimensional Problems

PCA helps address the curse of dimensionality by:

• Removing redundant features. Correlated variables are compressed into fewer components.
• Concentrating signal. True structure often lies in a lower-dimensional subspace.
• Improving distance-based methods. Reducing dimensions restores meaningful geometric relationships.
• Reducing overfitting. Fewer dimensions reduce the model’s capacity to memorize noise.

Summary

High-dimensional data introduces geometric and statistical challenges that degrade model performance. PCA provides a principled way to reduce dimensionality by using eigenvalues and eigenvectors to identify the directions of greatest variance. This reduces noise, improves stability, and restores meaningful structure for many machine learning algorithms.

General Machine Learning Workflow

Most machine learning projects follow a common sequence of steps, regardless of the specific algorithm used. A typical workflow includes:

1. Data Preparation
   Cleaning, transforming, encoding, and scaling predictor variables. Proper preprocessing ensures that models behave consistently and that no variable dominates due to scale or representation.
2. Train–Test Split
   Dividing the dataset into training and testing subsets. The training set is used to fit the model, while the test set provides an unbiased estimate of performance.
3. Model Training
   Fitting the model to the training data. Depending on the algorithm, this may involve estimating parameters, computing distances, or learning decision boundaries.
4. Hyperparameter Tuning
   Selecting algorithm settings (e.g., number of neighbors, tree depth, regularization strength) using cross-validation to avoid overfitting.
5. Model Evaluation
   Assessing performance using appropriate metrics:

• classification → accuracy, sensitivity, specificity, precision, F1, kappa,
• regression → MAE, RMSE, .

6. Model Interpretation and Validation
   Understanding model behavior, checking assumptions, and verifying that results generalize to new data.
7. Deployment and Monitoring
   Integrating the model into production workflows and monitoring performance over time.

This workflow applies broadly across supervised learning methods and provides a structured approach for building reliable, interpretable models.

Conclusion

This document provides a concise foundation for understanding core machine learning concepts that apply across many algorithms. By establishing a clear framework for variable types, encoding, scaling, distance functions, and evaluation metrics, we create a reusable reference that supports more advanced, algorithm-specific technical notes.

Subsequent documents, such as those covering k-Nearest Neighbors (k-NN), linear regression, decision trees, and ensemble methods will build directly on these foundational principles.

Don’t be frugal with the Google…

© MODBA — All rights reserved.