Support-based clustering has recently drawn plenty of attention because of its applications in solving the diffcult and diverse clustering or outlier detection problem. Support-based clustering method undergoes two phases: finding the domain of novelty and doing clustering assignment. To find the domain of novelty, the training time given by the current solvers is typically quadratic in the training size. It precludes the usage of support-based clustering method for the large-scale datasets. In this paper, we propose applying Stochastic Gradient Descent framework to the first phase of support-based clustering for finding the domain of novelty in form of a half-space and a new strategy to do the clustering assignment. We validate our proposed method on the well-known datasets for clustering to show that the proposed method offers a comparable clustering quality to Support Vector Clustering while being faster than this method.