BERTDOC: A Context-Aware Document Classification System Using BERT

Abstract

In this paper, the domain-specific classification scheme of documents has been suggested based on the Park BERT model (Bidirectional Encoder Representations from Transformers) to classify structured business documents, including invoices, purchase orders, and reports. The established systems of document classification are limited by poor scalability and high computational rate when applied in the setting of large organizations, which aggravates the increase in their vulnerability to errors. To discourage these shortcomings, a transformer-based solution is recommended, according to which the contextual semantics of the textual data is used to improve classifying performance. The suggested model presupposes textual input preprocessing followed by the tokenization of the documents with the help of the BERT tokenizer and fine-tuning of a pre-trained BERT model that is offered with several built-in classes to represent a company. The further assessment is realized with the help of the standard performance metrics such as accuracy, precision, recall, and F1-score. The comparative analysis of the results achieved in relation to the traditional machine learning methods shows a visible enhancement in the situational understanding and integrity in term of secondary categorization. The article clarifies the effectiveness of transformer-based structures in a practical paper administration framework and proposes future developments with the addition of more diverse data-sets and finer techniques of natural language processing.