Developing computational intelligence algorithms for learning insights from data has been a growing intellectual challenge. Much advances have already been made through data mining but there is an increasing research focus on deep learning to exploit the massive improvement in computational power. This chapter presents recent advancements in deep learning research and identifies some remaining challenges as drawn from using deep learning in the application area of gene expression modelling. It highlights deep learning (DL) as a branch of Machine Learning (ML), the various models and theoretical foundations, its motivations as to why we need deep learning in the context of evolving Big Data, particularly in the area of gene expression level classification. We present a review, and strengths and weaknesses of various DL models and their computational power to specific to gene expression modeling. Deep learning models are efficient feature selectors and therefore work best in high dimension datasets. We present major research challenges in feature extraction and selection using different deep models. Our case studies are drawn from gene expression datasets. Hence we report some of the key formats of gene expression datasets used for deep learning. As ongoing research we will discuss the future prospects of deep learning for gene expression modelling.