Investigation of the Effectiveness of Anomaly-Monitoring Methods for Network Intrusions in Ensuring the Information Security of Telecommunication Networks Using Autoencoders

Abstract

Ensuring the cybersecurity of telecommunication networks presupposes the detection of anomalies in traffic as a key element of protection systems. The purpose of the study was to assess the effectiveness of autoencoders for monitoring anomalies in network traffic and increasing the security level of telecommunication systems. The methodology was based on methods of comparative, systems and theoretical-analytical analysis to evaluate the effectiveness and optimise models of anomaly detection in telecommunication networks. It was established that telecommunication systems require adaptive traffic-analysis algorithms, since signature-based methods lose effectiveness in the face of new threats. Z-score and regression analysis are simple but do not take into account non-linearity and correlations between parameters. K-means and Density-Based Spatial Clustering of Applications with Noise are capable of identifying unknown patterns without data labelling, but these algorithms are sensitive to parameters and unstable with large data volumes. The Local Outlier Factor and Isolation Forest algorithms are effective for detecting rare events without prior training, but often generate too many false positives. It was found that autoencoders detect anomalies on the basis of reconstruction error, which makes it possible to identify unknown threats without the need for data labelling. Recurrent autoencoders take temporal dependencies into account, which makes it possible effectively to recognise long-term attacks such as Distributed Denial of Service or Botnet, but such models require significant computational resources. The classical accuracy indicator is insufficient; in Intrusion Detection System environments, Precision, Recall, F1-score, Receiver Operating Characteristic – Area Under the Curve and False Positive Rate are considered the main measures, as these metrics reflect the real effectiveness of attack detection. Combining autoencoders with clustering or statistical methods increases effectiveness, reduces false alerts and forms the basis of adaptive cyber defence. The practical value of the study lies in the possibility of using the results obtained to build high-performance systems for monitoring network traffic that are capable of timely detection of new and hidden cyberthreats.