Comprehensive Report on Real-Time Detection of Hidden Communication Channels in Network Traffic

Abstract

Covert channels in network communications pose a serious security challenge by enabling unauthorized, hidden data exchanges that bypass traditional detection mechanisms. These channels exploit subtle manipulations within network protocols such as packet timing, packet length, and header fields to secretly transfer information without raising suspicion. This dissertation explores the use of machine learning (ML) as an advanced and adaptive approach to detecting covert channels within network traffic. By leveraging diverse feature sets derived from network packets—including statistical measures of packet inter-arrival times, length distributions, and protocol header anomalies—this study aims to design robust, scalable detection models. The research investigates multiple machine learning algorithms such as Support Vector Machines, Random Forests, Decision Trees, and Deep Learning models including Autoencoders and Long Short-Term Memory (LSTM) networks to identify covert communication patterns effectively.

Datasets containing both legitimate and covert channel traffic are created or simulated using various covert channel techniques including timing-based and packet-length based channels. Extensive experiments are conducted to evaluate the detection accuracy, false positive rates, and model robustness against evolving covert strategies. Results demonstrate that ensemble learning methods and deep neural networks achieve detection accuracies exceeding 98%, significantly outperforming traditional rule-based and statistical approaches. Furthermore, explainable AI methods are incorporated to provide transparency and interpretability of the model decisions, aiding network administrators in understanding and responding to detected threats.

This dissertation also discusses architectural design for practical deployment of ML-enabled covert channel detection systems, covering data collection, preprocessing pipelines, feature engineering, model training, and real-time alerting. Future work directions include exploring generative adversarial networks (GANs) for simulating sophisticated covert channels, adapting models for IoT and emerging network paradigms, and integrating detection with automated response systems. Overall, the study establishes a comprehensive foundation for leveraging machine learning to enhance network security by detecting covert channels effectively and adaptively.