End-to-end solution with clustering method for attrition analysis

Abstract

We study a general attrition problem using unsupervised clustering and statistical approaches. The studied problem comes from retention problem in service industries. Our research provides an end-to-end solution from identifying hot job category to analyze the effectiveness of an incentive program applied to the selected categories. One of the barriers of studying the attrition problem is the lack of detailed features of an individual employee due to the confidentiality restriction. Different from the typical attrition approach that requires detailed individual information, we only use the aggregated attrition data and the internal business need data as the base, and cluster the job categories to give a recommendation. We converted the clustering results in a score for the recommendation. To avoid the monthly fluctuation, we apply exponential decay moving average multiple neighboring months on the snapshot scores to ensure consistent recommendation. The end-to-end solution also includes the impact analysis. By comparing the two general groups, we apply an approach similar to A/B test. We score the selected job categories with an effective score. We can apply this research to large consulting/service companies, and government agencies. For those enterprises or institutes, attrition avoidance is a major consideration as their main assets are their top performance employees. There also exist well-defined job roles and skill categories allowing to us to apply this approach.