Fuzzy Kernel Stochastic Gradient Descent Machines

Stochastic Gradient Descent (SGD) based method offers a viable solution to training large-scale dataset. However, the traditional SGD-based methods cannot get benefit from the distribution or geometry information carried in data. The reason is that these methods make use of the uniform distribution over the entire training set so as to sample the next data point for updating the model. We address this issue by incorporating the distribution or geometry information carried in the data into the sampling procedure. In particular, we utilize the fuzzy-membership evaluation methods which allow transferring the distribution or geometry information carried in the data to the fuzzy memberships. The fuzzy memberships is then normalized to a discrete distribution from which the next data point is sampled. This allows the training staying more focused on the important data points and tending to ignore the less impact data points, e.g., the noises and outliers. We validate the proposed methods on 8 benchmark datasets. The experimental results show that the proposed methods are comparable with the standard SGD-based method in training time while offering a significant improvement in classification accuracy