Open-Set Intrusion Detection and Semantic Analysis of Zero-Day Network Attacks Using Deep Reinforcement Learning and Large Language Models

by Shreyash Tiwari

The increasing prevalence of zero-day cyberattacks presents a significant challenge for modern Intrusion Detection Systems (IDS), as previously unseen network traffic often falls outside the distribution of supervised training data. Traditional deep learning-based IDS solutions operate under a closed-set assumption, requiring all attack categories to be known during training.

Consequently, novel attacks may be misclassified as benign traffic or incorrectly assigned to known attack classes, reducing the effectiveness of network defense mechanisms. Although recent open-set recognition approaches improve the detection of unknown traffic, they typically provide limited insight into the behavioral characteristics of detected anomalies, requiring additional manual investigation by security analysts.

This thesis presents a unified framework for open-set intrusion detection and semantic analysis of unknown network traffic by combining deep learning, reinforcement learning, and large language models. The proposed framework employs a Convolutional Neural Network (CNN) to learn feature representations from known network traffic classes and a Deep Q-Network (DQN) that uses SoftMax-derived uncertainty metrics, including maximum probability, probability gap, and Shannon entropy, to dynamically distinguish known from unknown traffic without manually defined confidence thresholds. To improve interpretability, a selective Large Language Model (LLM) reasoning module is applied only to traffic identified as unknown by the CNN-DQN pipeline. Unlike existing approaches that focus solely on detection or apply computationally expensive LLM reasoning across all traffic, the proposed framework combines efficient open-set detection with targeted semantic interpretation of suspicious network behavior.