A Comparative Analysis of MLP and LSTM for Defaulter Detection Using BERT and Custom Embeddings

Abstract

Predicting bank loan defaults is essential for financial institutions to reduce risk and maintain credit stability. Traditional models often rely on structured data, neglecting valuable insights from unstructured transaction text. In this work, we utilize deep learning to enhance prediction accuracy by converting transaction text into embeddings using two approaches: pre-trained BERT and a custom scratch-based technique developed without relying on any existing language models. These embeddings are used to train two architectures.Multilayer Perceptron (MLP) and Long Short-Term Memory (LSTM). The results show that the scratch-based embeddings consistently outperform pre-trained BERT embeddings across all performance metrics, achieving up to 95% accuracy with MLP and 89% with LSTM. In contrast, BERT-based models reached 75% and 79% accuracy with MLP and LSTM, respectively. To interpret model decisions and gain transparency, we apply SHAP (SHapley Additive exPlanations), identifying the most influential features contributing to correct predictions. Our findings demonstrate that custom embeddings tailored to domain-specific datasets can yield better results than generalized models. Additionally, integrating explainable AI techniques enhances trust and usability in financial applications. This work supports the use of deep learning and interpretable models for robust, high-accuracy loan default prediction using transactional text data.