Support Vector Machine (SVM) and Support Vector Data Description (SVDD) are the two most successful kernel methods for two-class and one-class classifications. This research primarily concerns a multi-modal approach to SVM and SVDD. A set of hyperplanes or hyperspheres was nominated in order to learn from the data. It was shown that multi-modal models work well with multi-distribution data sets. This research also investigated and proposed some variations of SVM and SVDD to boost their classification performance in the feature and input spaces. A way to link learning vector quantisation principle and margin maximisation principle which is crucial for kernel methods was proposed. The contribution of this thesis to the classification problem is as follows: 1. A Multi-Sphere Approach for SVDD which includes three models, e.g. hard model, fuzzy model and deterministic model for both one-class and two-class cases. 2. The Application of Multi-Sphere SVDD to clustering analysis problem which leads to Multi-Sphere Support Vector Clustering. 3. A Multi-Hyperplane Approach for SVM which includes hard, fuzzy and deterministic models. 4. Maximal Margin Learning Vector Quantisation. 5. The approaches to enhance SVM and SVDD for each subspaces are as follows: - Unified Support Vector Machine. - Fuzzy Approach to SVM,SVDD,SSLM and SS2LM. - Refinement Models for SVM and SVDD. - Small Sphere, Two Large Margins.