A Systematic Review of Hybrid Symbolic–Neural Models for Protocol Misuse Detection

Abstract

Protocol misuse detection has become a critical area in cybersecurity due to the growing complexity of network protocols and the rise of sophisticated attacks exploiting protocol-level vulnerabilities. Traditional intrusion detection systems rely on either symbolic rule-based approaches or data-driven neural models, both of which have notable limitations. Symbolic methods offer interpretability and precise rule enforcement but lack adaptability to novel threats, whereas neural models provide strong pattern recognition yet function as black-box systems with limited explainability. To overcome these challenges, hybrid symbolic–neural architectures have emerged as a promising solution, combining deep learning with rule-based reasoning to enhance both accuracy and transparency. This review synthesizes findings from multiple studies, focusing on architectural designs, optimization strategies, security models, and application domains. It categorizes approaches into neuro-symbolic intrusion detection systems, graph-based hybrid models, rule-enhanced deep learning frameworks, and reinforcement-driven adaptive detection methods. The findings highlight a growing emphasis on explainable AI, where symbolic reasoning is integrated with convolutional, recurrent, and graph neural networks to reduce false positives and improve interpretability. Despite these advancements, challenges such as scalability, real-time deployment, dataset imbalance, and lack of standardized benchmarks persist, indicating important directions for future cybersecurity research.