A Hybrid Machine Learning Approach for Real-Time Fraud Detection in Online Payment Transactions

Abstract

Fraud detection in online payment systems is a critical challenge due to the increasing volume of transactions and the sophistication of fraudulent activities. This paper presents a hybrid machine learning model that combines an autoencoder for unsupervised feature extraction with Gradient Boosting for fraud classification. The proposed model achieves high accuracy and computational efficiency by leveraging dimensionality reduction to optimize processing and maintain scalability. Experimental results demonstrate the model’s robustness to imbalanced datasets, retaining precision and recall as the class imbalance increases. With an average prediction latency of 2.8 milliseconds per transaction, the hybrid model is suitable for real-time fraud detection in high-volume payment environments. While the model performs effectively, limitations such as a lack of adaptability to evolving fraud patterns and limited interpretability are identified. Future work will focus on integrating adaptive learning and explainable AI techniques to enhance the model’s transparency and responsiveness, setting the stage for more advanced fraud detection frameworks.