With the advance of the communication technology and the massive flow of information across the internet, it is becoming urgent to keep the confidentiality of the transmitted information. Using the internet has been extended to several fields such as e-mail, e-commerce, e-learning, health and medicine, shopping, and so on. Cryptography is the study of different techniques for securing the communication between the sender and the receiver. One of the most known cryptosystems is Merkle–Hellman Knapsack Cryptosystem (MHKC). It is one of the earliest Public Key Cryptosystem (PKC) that is used to secure the messages between the sender and the receiver. Developing a powerful cryptosystem comes after studying the fragility points of the current cryptosystems. The Whale Optimization Algorithm (WOA) is one of the most recent nature-inspired meta-heuristic optimization algorithms, which simulates the social behavior of humpback whales. WOA has validated excellent performance in solving the continuous problems and the engineering optimization problems. This paper introduces a novel Modified version of WOA (MWOA) for cryptanalysis of MHKC. The sigmoid function is used to map the continuous values into discrete one. A penalty function is added to the evaluation function to deal with the infeasible solutions. The mutation operation is employed for improving the solutions. The results show that MWOA is more effective and robust than other algorithms in the literature.