AbstractsComputer Science

Internet is becoming the most indispensable component in our daily life. The innovation of new services, e.g., social network applications, blogs and twitter, video sharing, etc., have completely changed the way how people think and behave. As a consequence, staying connected is the primary demand from most of the people. To fulfill such a demand, various operational networks are established, such as DSL networks and cellular networks, to provide customers with Internet access anywhere and anytime. While network service providers are enjoying the business opportunity from such an increase of demand, they also face the challenge that the increase of human expert knowledge cannot meet the management requirements due to the quick expansion of such large-scale networks. An efficient way of managing such a network is crucial for minimizing operational cost, improving customers' experience and hence reducing churn, i.e., customers quitting the service. In this thesis, we conduct a systematic study of applying advanced statistical machine learning techniques to solve two representative network management problems: traffic classification and troubleshooting in a large operational DSL network, which consists of millions of users and tens of millions of basic devices. We present the design of three statistical machine learning based systems to solve these two problems, while meeting the operational constraints and requirements in such a network. In particular, we design FLOWCLASS, a light-weight flow-level traffic classification system. FLOWCLASS is characterized with a modular architecture, which combines a series of simple linear binary classifiers each of which can be efficiently implemented and trained on vast amounts of flow data in parallel, and integrates them in such a manner that it attains the accuracy of more sophisticated classifiers. To handle the application scenarios when FLOWCLASS is not applicable or not accurate, we propose TAGCLASS. TAGCLASS incorporates a novel set of features  – the spatial distribution of traffic classes in the network, which is represented by colored traffic activity graphs, and employs a two-step model. In the first bootstrapping step, traffic is classified based on solely the associated traffic attributes. In the second calibration step, the results from the bootstrapping step are corrected or reinforced based on the spatial relationships of different traffic classes in the colored TAGs. For the troubleshooting problem, we propose NEVERMIND, a proactive solution to troubleshoot DSL customer problems. NEVERMIND contains two main components: ticket predictor and trouble locator. Ticket predictor detects potential problems which may lead to future customer tickets, and trouble locator prioritizes potential problem locations to assist technicians in diagnosing problems. Innovate techniques such as top-N average precision based feature selection and combined hierarchical models are introduced to adapt existing statistical learning techniques to achieve good accuracy with…