A proposed statistical model for spam email detection

The keyword list-based spam email detection system uses keywords in a blacklist to detect spam emails. To avoid detection, keywords are written as misspellings, for example «virrus», «vi-rus» and «viruus» instead of «virus». The system needs to update the blacklist from time to time to detect spam emails containing such misspellings. However it is impossible to predict all possible misspellings for a given keyword to add those to the blacklist. This paper proposes a statistical framework to solve this problem. A keyword is represented as a Markov chain where letters are states. A Markov model is then built for the keyword. In order to decide an unknown word as a misspelling of a given keyword, a statistical hypothesis test is used. Experiments showed that the proposed statistical models could achieve the detection error rate of 0.1%.