Flow Monitoring
Network traffic monitoring is an essential component of every modern network infrastructure. For the large ones, it is currently the best practice to use technology based on IP flows, which is exactly the scope of interest in our R&D activities.
Since we are responsible for traffic monitoring of the national research and education network perimeter , we have to deal with state-of-the-art high bandwidth data networks with communication speeds exceeding 100Gb/s. Therefore, we develop unique hardware cards with HANIC firmware that are accelerating the processing and aggregating flow records inside monitoring probes – meters, also called flow exporters. Our monitoring probes can operate with network lanes reaching up to 200Gb/s speeds, and our current research goal targets 400Gb/s. As a software part of the monitoring probes, we use flowmonexp by Flowmon Networks (Kemp), and for experiments we use our own open source IP Flow exporter ipfixprobe.
Exported IP flow data have to be processed and stored somewhere. Naturally, the monitoring infrastructure of the ISP size networks has to process many flow records per second (in our case, cca 150K flows/s on average, and about 300K flows/s in peaks). Therefore, besides fast monitoring probes, we need a high-speed flow collector as well. For this purpose, we have developed an open source collector called ipfixcol2 (which is the second and enhanced generation of ipfixcol collector). This collector is able to collect flow data from all 6 monitoring probes (some of them monitor multiple lines) and stores them using a distributed system that has been developed in SecurityCloud project. This distributed approach significantly increases the speed of data queries, e.g., during the incident handling.
Besides flow data storage, we also have deployed NEMEA system for stream-wise traffic analysis and detection of suspicious communication. NEMEA is one of the main sources of detected security events that are reported to the incident sharing system Warden (shares security events among the community of network operators) and reporting system Mentat (automatically produces notifications to the users). From the research point of view, NEMEA is used to create prototypes of processing and detection algorithms. Currently, we are focused on encrypted traffic analysis using IP flow data extended with packet-level information.