In this paper we consider a portfolio selection problem under the global minimum variance model where the optimal portfolio weights only depend on the covariance matrix of asset returns. First, to reflect the rapid changes of financial markets, we incorporate a time-varying factor in the covariance matrix. Second, to improve the estimation of the covariance matrix we use the shrinkage method. Based on these two key aspects, we propose a framework for shrinking the time-varying inverse conditional covariance matrix in order to enhance the performance of the portfolio selection. Furthermore, given the shortcoming that the inverse covariance matrix is inaccurate in a number of cases, we develop a new method that transforms the inverse of the covariance matrix into a product to improve the performance of the inverse covariance matrix, and prove its theoretical availability. The proposed portfolio selection strategy is applied to analyze real-world data and the numerical studies show it performs well.