An EM-based Ensemble Learning Algorithm on Piecewise Surface Regression Problem

Abstract

A multi-step Expectation-Maximization based (EM-based) algorithm is proposed to solve the piecewise surface regression problem which has typical applications in market segmentation research, identification of consumer behavior patterns, weather patterns in meteorological research, and so on. The multiple steps involved are local regression on each data point of the training data set and a small set of its closest neighbors, clustering on the feature vector space formed from the local regression, regression learning for each individual surface, and classification to determine the boundaries for each individual surface. An EM-based iteration process is introduced in the regression learning phase to improve the learning outcome. In this phase, ensemble learning plays an important role in the reassignment of the cluster index for each data point. The reassignment of cluster index is determined by the majority voting of predictive error of sub-models, the distance between the data point and regressed hyperplane, and the distance between the data point and centroid of each clustered surface. Classification is performed at the end to determine the boundaries for each individual surface. Clustering quality validity techniques are applied to the scenario in which the number of surfaces for the input domain is not known in advance. A set of experiments based on both artificial generated and benchmark data source are conducted to compare the proposed algorithm and widely-used regression learning packages to show that the proposed algorithm outperforms those packages in terms of root mean squared errors, especially after ensemble learning has been applied.