Graph-based Detection of Malicious Network Communities

Abstract

In this thesis, we use graph based methods in conjunction with behavioral modeling to uncover hidden malicious communities and peer-to-peer tra c. The nature of malicious tra c, and its tendency to rally in order to communicate with its owner opens a possibility to detect malicious tra c by revealing hidden sub-structures of network tra c. In fact, besides discovering the presence of an infection, analyzing network tra c also enables inference of valuable context information about the malicious campaign as a whole, often leading to a more precise attribution than is possible using only a hostbased solution. In this work, we focus on the detection approaches that observe the hidden structures and exploit them to uncover malicious command & control (C&C) servers. Peer-to-peer (P2P) protocol is a popular choice with malware authors to be used as a C&C channel. Therefore, we propose a uni ed solution to identify P2P communities operating in a monitored network. We propose an algorithm that is able to 1) progressively discover hosts in the monitored network that cooperate in a P2P network and to 2) identify that P2P network. Starting from a single known host, other hosts participating in the P2P network are identi ed through the analysis of widely available and standardized IPFIX (NetFlow) data. It is able to identify a large range of both legitimate and malicious P2P networks, is highly scalable and the use of standard meta-data without access to tra c content makes it easy to deploy and justify from privacy protection perspective. Even malware families that do not rely on a P2P-based C&C channels resort to highly dynamic C&C structures to counter security industry approaches based on blacklisting known malicious domains. It is therefore important to automatically follow the migration of C&C servers. We propose to use a well-known Probability Threat Propagation (PTP) with a novel graph representation capturing connections from clients to servers. The proposed graph representation is highly condensed, preserves privacy, allows us to nd malicious domains that cannot be found using existing graph representations and is harder to evade by malware authors. We propose two behavioral models for HTTP tra c together with kernel-based similarity and distance functions that can be conveniently used to extend the ndings of PTP. For any domain marked as malicious by PTP we can nd other domains with identical or similar behavior, which are likely also malicious. This signi cantly increases the number of discovered malicious domains. All proposed algorithms and representations are veri ed using extensive data sets spanning hundreds of independent networks. The validity of proposed approaches was further veri ed in a large-scale deployment within the Cisco Cognitive Threat Analytics.