In the minimum variance model, the covariance matrix plays an important role because it measures the risk and relationship of asset returns simultaneously under the normality assumption. However, in practice, the distribution of asset returns is nonnormal and has an obvious fat-tail nature. In addition, the risk is one-sided. In this paper, the main objective is to propose a better tool to replace the covariance matrix. The covariance matrix can be decomposed into two parts: a diagonal variance matrix and a square matrix with its elements being the Pearson correlation coefficient. A substitution of the covariance matrix is presented by replacing the variance and Pearson correlation coefficient in the decomposition of the covariance matrix with a semivariance and distance correlation coefficient, respectively. The proposed portfolio optimization strategy is applied to empirical data, and the numerical studies show the strategy performs well.