Support Vector Machines
- Requires solving a Quadratic Programming problem which can be computationally intensive
- Finding the right kernel function, for linear classification of data required by SVMs, is non-trivial task.
- Assuming a kernel function and optimizing the cost function are done as different steps (Neural Networks, where these are done simultaneously poses this as its advantage over SVMs)
Perceptron (with FLD)
- Requires the data to be linearly separable. If the classification accuracy of the perceptron method is bad, kernel methods (eg. SVMs) might be required.
- If the required class means and covariances are not known, they can be estimated from the training set. Parameter estimation methods like maximum likelihood estimate or the maximum a posteriori estimate may be used
- Regularization might be required (for finding the inverse) to avoid overfitting issues.
KNN Classification
- This classification method gives very good results if huge training data is available.
- For a given input data, with no apriori knowledge, choosing appropriate distance metric is very important. Distance metrics are used in density estimation methods (Parzen windows), clustering (k-means) and instance based classification methods (Nearest Neighbors) etc. Euclidean distance is used in most of the cases, but in cases where the relationship between data points is non-linear, selection of a distance metric is a challenge. Here is a reference addressing this issue: citeulike
