Bei's Study Notes

Machine Learning Study Note - Introduction
Last updated: 2017-10-10 15:39:14 PDT.

Types of Machine Learning Problems (wiki)

ML tasks can be classified into three broad categories based on the nature of learning signal or feedback avialable to a learning system:

  1. Supervised learning
  2. Unsupervised learning
  3. Reinforcement learning

Between supervised and unsupervised learning is semi-supervised learning.

Depending on the output:

  1. Classification
  2. Regression
  3. Clustering
  4. Ranking
  5. Recommendation


  1. Density estimation
  2. Dimensionality reduction


  1. Paramatric methods - Methods that make assumptions that the data comes from a distribution and it intends to estimate these parameters.
  2. Nonparamatric methods - Methods that does not assume certain form of distributions. (Or rather, a free-form of distributions. "nonparametric really means many parametric.").

Machine Learning Approaches

  1. Linear Model
  2. Kernel machines
  3. Decision tree
  4. Bayesian Estimation
  5. Hidden Markov Models
  6. Ensemble learning
    1. Random Forest
    2. Gradient Boosting
  7. Graphical models (Bayesian networks)
  8. Clustering
  9. Association rules learning
  10. Artificial neural network
  11. Reinforcement learning
  12. Generalized Additive Model
  13. Inductive logic programming
  14. Feature learning
    1. Sparse dictionary learning
  15. Genetic algorithms
  16. Rule-based machine learning
  17. Learning classifier systems (LCS)

Model Selection

Hypotheses set \mathcal H . Bias - underfitting. Variance - overfitting. Generalization. Triple trade-of.

Cross validation. Test set. Analogy with exams.