FlowMon Probe Handbook

The FlowMon Probe is a passive network monitoring device based on the COMBO6(X) technology. It is able to collect dynamic data about IP flows and export them to external collectors in the NetFlow (version 5 and 9) and IPFIX (IP Flow Information eXport) format. The probe has great contribution to the safety and reliability of your network. You can obtain information about attacks and data transfers going in and out of your network. The probe is remotely configurable using web or terminal interface.

Figure 1.1. FlowMon Probe Monitoring Principle

So far, NetFlow data are mostly generated by IP routers. In contrast, our NetFlow probe is designed as an autonomous device working essentially as a T-splitter: when inserted into a network link, the incoming traffic is passed directly to the original destination and a separate copy of the link data is processed by the probe in parallel. From the network perspective, the probe can be classified as a repeater that is invisible at both the network and link layer.

Using a specialized probe for gathering data about IP flows has several advantages over the traditional routerbased setup:

In contrast, standalone monitoring probe is essentially a stealth device - invisible at both Layer 3 and 2 - dedicating all its resources to the tasks of flow record acquisition and processing.

Figure 1.2. NetFlow Principle

FlowMon package is available via WWW download page: http://www.liberouter.org/clients

To verify that the downloaded files are genuine and complete SHA1 digests are available. Download both the CHECKSUM.SHA1 file and the package file to the same directory. Run the sha1sum(1) command to verify the package file e.g. flowmon-1.4.0.tgz:

$ cat CHECKSUM.SHA1 | grep flowmon-1.4.0.tgz | sha1sum -c
      

If there are any errors, they will be reported.

	/firmware      - COMBO6(X) card firmware (*.mcs files)
	/base          - source code for software tools and drivers
	  /mk                        - build system (makefiles)
	  /sys_sw/drivers            - kernel drivers
	  /sys_sw/hwtools            - necessary hardware tools for COMBO6(X) card
	  /sys_sw/lib*               - libraries necessary for other tools
	  /sys_sw/projects/flowmon   - exporters, testing programs, configuration
                                       and documentation for using FlowMon
	  /sys_sw/projects/netconf   - NETCONF protocol programs
	  /sys_sw/swtools/csxtool    - tool for handle COMBO6(X) XML files
	/doc           - FlowMon project documentation - FlowMon Handbook
	ERRATA         - list of known bugs and their solutions
	README         - short manual how to build, install and use FlowMon Probe
	RELNOTES       - differences against previous release
	  

There are two possibilities how to use FlowMon probe - either locally by flowmonlkm(1) and flowmon(1) programs (described in Chapter FlowMon Probe - local using) or remotely via remote command line interface or Web frontend (described in Chapter Using FlowMon Probe - remote configuration).

This package can be used with the COMBO6(X) cards described in the following table. COMBO6(X) cards have to have installed PCI bridge version c610.04.07 or higher (you can find details about PCI bridge and its version at http://www.liberouter.org/documents/pci_bridge_upgrade.php).

Figure 2.1. COMBO6X Card

Supported firmware depends on the type of card you use.

Firmware supports processing of Ethernet and all basic IPv4 and IPv6 headers:

FlowMon software works on GNU/Linux OS with 2.4 and 2.6 kernels. The NetFlow v5, v9 and IPFIX protocols are supported. The software has been tested on computers running Red Hat Enterprise Linux, CentOs, Ubuntu and Debian.

After plugging COMBO6(X) card into your PCI slot, you should test connection between the card and your PC. We use lspci(8) utility for this purpose. lspci(8) is a utility for displaying information about all PCI buses in the system and all devices connected to them. For correct recognition of the COMBO6(X) card you need update PCI ID Database used by lspci(8) or download pciutils-2.2.2 (program collection containing lspci(8) or later. If the lspci(8) output contains the following line your COMBO6(X) card is connected properly.

	$ lspci -d 18ec:
	03:01.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6X (rev 01)
	04:02.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6 (rev 01)
	

There are three possibilities, how to insert FlowMon probe to the network - you can connect FlowMon probe at a mirror port of some network device, utilize network tap or insert into a line as a repeater.

FlowMon Probe at a Mirror Port

If you want to check FlowMon features you can simply mirror traffic from your router to the FlowMon probe.

Figure 2.2. FlowMon Probe Inserted at Mirror Port

FlowMon Probe Connected via Network Tap

Another way of how to connect FlowMon probe in your network is to utilize network tap, e.g. optical splitter (see figure below).

Figure 2.3. FlowMon Probe Connected via Optical Splitter

FlowMon Probe Inserted in a Line

In this case the FlowMon probe works as a T-splitter: when inserted into a network link, the traffic is passed directly to the original destination and a separate copy of link data is processed by the probe in parallel. From the network perspective, the probe can be classified as a repeater that is invisible at both the network and link layer. Description of this repeater mode is in the chapter Repeater.

Figure 2.4. FlowMon Probe Inserted in Line as a Repeater and Sending Data to Collector

FlowMon Probe Port Numbers

The general rule for numbering card ports is that the ports closer to the motherboard (PCI slot) have lower numbers, e.g. the closest port has number 0, the next one number 1 etc. See figures bellow for examples:

Figure 2.5. FlowMon Probe COMBO-4SFPRO Card Port Numbers

Figure 2.6. FlowMon Probe COMBO-2XFP2 Card Port Numbers

pkgtool(1) helps to build, install and uninstall the FlowMon distribution package.

	$ tar -xzvf flowmon-XX.YY.ZZ.tgz
	$ cd flowmon-XX.YY.ZZ/base
	$ ./pkgtool --build
	

If this is the first Liberouter package you have ever installed, you can define installation directory with the --prefix=path option. But remember that this installation path MUST not exists (e.g. /usr/local is invalid installation path on most systems because this directory exists). This restriction is due to new installation framework which enables easy package uninstall or package switching. More information about these features can be found in the liberouterpkg section.

The --prefix option takes effect only during building package. If no prefix is set then /usr/local/liberouter path is used.

	$ ./pkgtool --build --prefix=/usr/local/flowmon
	

If you have previously installed some Liberouter package (nific, nic, ids, flowmon with liberouterpkg mechanism), the installation path is detected automatically as path used for the first installed package.

Remember that installation path given as --prefix parameter will contain next subdirectories for binaries, libraries, man pages, etc. These directories can be affected by future uninstalling or package switching so it could be used as installation directory only for Liberouter packages.

Installed tools are divided to three parts.

Figure 2.7. Scheme of Installation Types

You can use more than one type of installation (typically server and web) by specifying keywords separated by comma(s)

	# ./pkgtool --install=server,web
	

All FlowMon Probe tools come with its manual pages so for information about any tool (included configuration files) you can see these man pages by man(1) program, e.g.

        $ man flowmon_nf5
        $ man flowmon.conf
        

There are necessary following post-install steps:

The flowmon-1.3.0 is the first Liberouter package using liberouterpkg tool enabling package switching and fully package uninstalling. If you have installed any previous Liberouter package without liberouterpkg, please remove it completely (including all libraries and drivers) to ensure proper behavior of the installed package.

More information about these features can be found in the liberouterpkg section.

If you are installing new version of previously installed package (e.g. you have installed flowmon-1.3.0 and now you are installing flowmon-1.3.1 package) you will be asked by pkgtool(1) to decide if you wish to keep your own (but may be obsolete) configuration files or to overwrite them with our default (but up-to-date) configuration files.

liberouterpkg script is new tool covering new Liberouter package installation framework which enables safe and easy package uninstalling or simple package switching (and using different project packages on the same PC). To display all available functions of the liberouterpkg script use --help option

    $ liberouterpkg --help
    

liberouterpkg uses /etc/liberouter/packages.list configuration file that stores information about installed packages.

      # liberouterpkg --list
      # Installed packages:
      flowmon-1.3.0
      

      # liberouterpkg
      flowmon-1.3.0
      

      # liberouterpkg --uninstall=flowmon-1.3.0
      

      # liberouterpkg --switch=ids-1.0.0
      

This section describes flowmon.conf(5) configuration file placed in the /etc/liberouter/ directory. This configuration file is read by FlowMon Probe starting ( flowmonlkm(1), flowmon(1)) and controlling ( flowmond(1)) programs.

flowmon.conf(5) is actually a list of definition of environment variables.

    VARNAME=VALUE
    

The file can contain blank lines or lines starting with '#' which are ignored by programs and can be used for comments.

You can freely modify this file and redefine default values of the probe settings. During installation of the next flowmon package version you will be inquired by pkgtool if you prefer to keep your own configuration file or to rewrite it with our up-to-date version. So you don't have to be worry to lose your changes. Up-to-date file is during installation process always copied to the $PREFIX/etc/liberouter (by default /usr/local/etc/liberouter) directory. So you can use this file as a backup copy.

Variable definitions are divided into several parts.

Remote configuration programs use as source of the configuration data set of configuration files in XML format (configuration datastores). These files are located in the /etc/liberouter/netconf/ directory by default. This location can be changed in the netconf.conf(5) configuration file.

FlowMon Probe remote configuration uses three configuration datastores called running, startup and candidate (concrete filenames representing these datastores are defined in the netconf.conf(5) configuration file).

Definition of the FlowMon Probe XML configuration scheme can be found at http://www.flowmon.org/flowmon-probe/devel/config/flowmon-rng/.

After building and installing package including post-install steps, all FlowMon tools (included man pages) are available as any other system tools. There are two main scripts to start the probe - flowmonlkm(1) and flowmon(1). Both scripts use flowmon.conf(5) configuration file.

        $ flowmon -h
        $ man flowmon
        

        $ dmesg
        combo6#0: device 0xf1010100 (NETFLOW_1Gbps_Probe) successfully attached
	

        user.*                  /var/log/user.log

        # flowmonlkm -l
	Loading FlowMon kernel modules for COMBO6X card.
	szedatax_c6pcr         13504  3
	szedata                31596  5 szedatax_c6pcr
	libermemalloc           4164  2 szedatax_c6pcr,szedata
	combo6x                13028  1 szedatax_c6pcr
	combo6                 20072  1 szedatax_c6pcr
	combo6core             24052  7 szedatax_c6pcr,combo6x,combo6
        $ flowmon -ec collector.liberouter.org:60000
        

        $ csid -s
        Board    : combo6
        Addon    : mtx2
        Chip     : xcv2000
        LAN ports: 4
        Firmware : ok
        SW       : 0xf1010002
        HW       : 0x00000006
        Text     : NETFLOW_1Gbps_Probe

        $ flowmon -s
        

        # flowmonlkm -r
        

	$  ps aux | grep flowmon
	flowmon   3023  12:19   0:11 flowmon_nf9 collector.liberouter.org 60000 -I 65535
	flowmon   3105  12:26   0:04 flowmon_nf5 -d localhost 3003
	

Startup scripts are used to start some service (or some script) at the machine boot time. Our sample startup script is used to automatically start up the FlowMon Probe (loads kernel modules if necessary, boots firmware and set up the probe behavior) anytime your PC is starting up. Sample script is stored in the package directory structure in the base/sys_sw/projects/flowmon/doc directory as a flowmon.rc file. It is prepared for use in the SysV init system (and tested on the Red Hat Linux distribution). Script is commented so you can get a lot of information directly from the script.

Startup script uses flowmon.conf(5) to start flow exporter(s).

        # ./flowmon stop
        

        # chkconfig flowmon off
        

        # chkconfig --del flowmon
        

        # ./flowmon status
	Loaded kernel modules for the FlowMon probe:
	netflow_ph1             7684  2
	szedata                21452  3 netflow_ph1
	libermemalloc           2948  2 netflow_ph1,szedata
	combo6                 13928  1 netflow_ph1
	combo6core             15956  4 netflow_ph1,combo6
	
	Running FlowMon exporter(s):
	flowmon_nf9 collector.liberouter.org 3002 -I 6553
        

To read data from HW and send them to collector you can run several instances of FlowMon exporters. Before running exporter you must start FlowMon probe (load kernel modules and run flowmon script).

FlowMon exporter has possibility to anonymize exported data, perform filtering, exporter sampling, use IPv4 or IPv6 transport etc. There are three versions of FlowMon exporter, flowmon_nf5(1) for exporting in NetFlow version 5 export format, flowmon_nf9(1) for exporting in NetFlow version 9 export format and flowmon_ipfix(1) for exporting in IPFIX export format.

	$ flowmon_nf5 -d collector.liberouter.org:60000
	$ flowmon_nf9 -d collector.liberouter.org:60001
	$ flowmon_ipfix -d collector.liberouter.org:60002

	$ flowmon_nf9 -c 0:0 -a aes:fields=src,dst -d collector.liberouter.org:60000 

The exporter program will start sending flow records to the configured collector. To test that it is really the case, you can run a packet sniffer such as tcpdump(1) or ethereal(1), for example

	# tcpdump -i interface 'udp dst port collector_port'
	# tcpdump -i eth0 'udp dst port 60000'
	tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
	listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
	12:23:41.656669 IP exporter.flowmon.org.32768 > collector.liberouter.org.60000: UDP, length 1424
	12:23:41.667760 IP exporter.flowmon.org.32768 > collector.liberouter.org.60000: UDP, length 1424
	12:23:41.667798 IP exporter.flowmon.org.32768 > collector.liberouter.org.60000: UDP, length 1420
	...

You should see the NetFlow traffic sent by the flow exporter to the collector. To test basic collector functionality you can use flowmoncol(1) tool.

	$ flowmon_nf5 -d localhost:60000
	$ flowmoncol -l 60000 -m 4
	seq 871320, dur 106 ms, in 0, proto 6, 195.113.123.86:36174 -> 82.208.7.33:80, 
	packets 17, octets 912, aps 53, bps 68830, pps 160
	seq 871320, dur 82 ms, in 0, proto 6, 147.231.249.66:3902 -> 77.75.72.72:80, 
	packets 4, octets 1689, aps 422, bps 164780, pps 48
	seq 871320, dur 40207 ms, in 0, proto 6, 195.113.168.70:49307 -> 80.239.235.195:443, 
	packets 11, octets 586, aps 53, bps 116, pps 0
	...

For further information please see exporters man pages:

	$ man flowmon_nf5
	$ man flowmon_nf9
	$ man flowmon_ipfix
	

Repeater duplicates data from PORT0 to PORT1 and vice versa. PORT2 and PORT3 are not utilized on COMBO-4SFP or COMBO-4MTX. COMBO-4SFPRO card is able to mirror traffic from PORT0 to PORT2 and from PORT1 to PORT3. Input streams from PORT0 and PORT1 are processed by IBUFs. In each IBUF Input Sampling rate can be set. Also, repeater is not present on COMBO2-XFP2 card.

It is necessary to state that the repeater is independent on the state of the rest of firmware and software as well. It is only dependent on state of the FPGA chip and thus on the state of the host computer (power supply, firmware booting). So if you have connected the FlowMon probe directly in the link as repeater (see section FlowMon Probe Inserted in a Line), the built-in repeater starts to transfer packets in both directions between interfaces 0 and 1, and at the same time mirror packets to interfaces 2 and 3. If you choose this option it is recommended to use UPS (uninterruptable power supply) so the repeater is available during power failures.

During reloading or resetting card, it takes about 4 seconds to start repeating again. Also monitoring starts 4 seconds later after reset or reload.

Figure 4.1. Scheme of the Repeater Connection

phyterctl(1) is tool used to display and change configuration of 4 interfaces available on COMBO-4SFPRO cards. The tool displays information about link status, resolved speed or duplex mode on link. phyterctl(1) is also able to change the advertised speed and duplex mode and provides r/w access to internal registers of the physical layer IC.

ibufctl(1) is used to display and change configuration of IBUF components in FlowMon COMBO6X designs.

	$ phyterctl -s100 -i0 ... advertise 100Mbps on interface 0
	$ ibufctl -s100 -i0 ... set 100Mbps input on interface 0

Example of phyterctl(1) listing with GBIC EEPROM information:

	$ phyterctl -c gbic
	Settings for card 0 (device /dev/combosix/0):
	------------------------------ Interface 0 ---
	Transceiver      FINISAR CORP.
	Model		 FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	------------------------------ Interface 1 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	------------------------------ Interface 2 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Link status      Down
	------------------------------ Interface 3 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	

More information can be found in the phyterctl(1) and ibufctl(1) man pages or in the README files placed in the base/sys_sw/hwtools/phyter/ and base/sys_sw/hwtools/ibufctl/ directories.

This section describes step-by-step process to prepare server (PC with installed COMBO6(X) card) and manager PC (any PC with Linux OS connected to the Internet).

        # ./pkgtool --install=server
        

          VARNAME=VALUE
        

          Port 830
          

              Port 22
              

          Subsystem       netconf /usr/local/liberouter/bin/netconf-agent