An enhanced autoencoder for anomaly detection in imbalanced financial fraud data

Abstract

The detection of anomalies in financial fraud is a complex task because of the imbalance in data and the wide variety of fraudulent methods being used. Traditional autoencoders often struggle with this imbalance, because they tend to be biased towards the majority class and have reduced effectiveness in detecting anomalies. To address these challenges, this study introduces an enhanced autoencoder model that aims to improve performance on imbalanced datasets by incorporating class-specific reconstruction losses and gradient clipping techniques. This research does not use the synthetic minority oversampling technique (SMOTE) method. The purpose is to avoid potential noise and bias caused by SMOTE. Also, without using SMOTE, the required computer capability is reduced and more focused on the model improvements. Key meansurements in this study are specificity, recall and ROC-AUC curve which can highlight ability of model in the imbalanced datasets. This thesis uses two distinct datasets—the Bank Account Fraud (BAF) dataset and the BankSim dataset to evaluate the proposed improvements. Our findings reveals significant enhancements in key performance metrics such as precision, recall and, F1-score, when compared with traditional autoencoders. These results highlight the potential of advanced deep learning techniques for enhancing anomaly detection systems. Overall, the main outcome of this research is the design of an enhanced autoencoder model that significantly improves the detection of fraudulent transactions in imbalanced datasets by addressing class imbalance without relying on synthetic oversampling, achieving better outcomes in specificity and recall, and differentiating minority group compared with traditional methods. By bridging theoretical advancements with practical evaluation, this research contributes to the development of more reliable and efficient fraud detection systems, while also offering a framework applicable to other domains, such as healthcare diagnostics and cybersecurity.

Date of Award	2025
Original language	English
Supervisor	Shuangzhe LIU (Supervisor)