A Comprehensive Optimisation Framework for Machine Learning-Based Big Data Processing in Distributed Intelligent Systems

Abstract

This paper investigates ways to improve the efficiency of machine learning algorithms for processing large datasets in distributed intelligent systems by identifying the optimisation configuration that simultaneously reduces training time, lowers resource consumption, and improves model quality. The study was conducted as a computational experiment in a simulated distributed environment using large datasets for classification, anomaly detection, transaction analysis, and time series tasks. Four optimisation scenarios were compared with a baseline configuration without optimisation: data structure optimisation, computational process optimisation, inter-node interaction optimisation, and combined optimisation. The baseline scenario achieved an average training time of 78.4 minutes, RAM usage of 27.4 GB, and classification accuracy of 0.887. Data structure optimisation reduced training time by 12.4%, decreased RAM usage by 9.1%, and increased classification accuracy by 1.8%. Computational process optimisation produced the best individual result, reducing training time by 24.7%, increasing processing speed by 21.3%, and improving classification quality by 2.6%. Optimisation of inter-node interaction reduced network load by 18.9% and total execution time by 15.6%. The combined optimisation scenario achieved the highest overall performance, reducing training time by 34.8%, lowering RAM usage by 14.2%, increasing classification accuracy by 3.1%, and improving the overall classification quality index by 3.8%. The findings demonstrate that the greatest efficiency in distributed intelligent systems is achieved through comprehensive optimisation of data structures, computational processes, and inter-node interaction, providing an effective approach for high-performance machine learning systems designed to process big data.