I found some useful recommendations for using non-linear discrimination techniques in the book "Statistical Pattern Recognition" by Andrew Webb. In this page, I will summarize the main points of the recommendations which I think are very interesting:
Non-liner discriminations methods for classification now are very popular and easy to implement. There are also many available sources of software both free or commerical for doing them. However, before using non-linear disrimination methods, we should consider if they are really neccessary and the applying them can really improve the performance. The first thing we need consider is the dicision boundary is realy non-linear. Someone after doing some linear discrimination and getting bad classification results, will move immediately to Neural network or Support Vector Machine with nonlinear kernel with the hope that the result will be significantly improved. However this is not always the case. If two classes are not seperable, even very complicated model doesn't help much. In these cases, what we need to do is to increase the dimension of feature vectors (additional variables).
So the recommendations are as follows:
1. Try simple pattern recognition technique (K-nearest neighbor, linear discriminant analysis)first as a baseline before trying Neural network.
2. Use a radial basis function RBF (unsupervied selection of centers, weights optimized using a square error criterion) to have a feel if the non-linear method can work?
3. A regularized solution for the weights of an RBF is used
4. For a model that provides approximations to the posterior probabilities that enable. changes of priors and costs to be incorporated into a trained model, an RBF is used.
5. Knowledge of the data generation process is used, including noise on the data, for network design or data preprocessing.
6. For classification problems in high-dimensional spaces where training data are representative of test conditions and misclassification rate is an acceptable measure of classifier performance, support vector machines are implemented.
