| Date: | February 2007 |
|---|---|
| Version: | 1.1.0 |
| Author: | The Liberouter Project Team |
| Organization: | CESNET, z.s.p.o. |
The FlowMon probe is a passive network monitoring device based on the COMBO6(X) technology. It is able to collect dynamic data about IP flows and export them to external collectors in the NetFlow version 5 and 9 format. The probe has great contribution to the safety and reliability of your network. You can obtain information about attacks and data transfers going in and out of your network.
You can check latest news about FlowMon Probe at our web sites:
FlowMon package is available via WWW download page:
To verify that the downloaded files are genuine and complete SHA1 digests are available. Download both the CHECKSUM.SHA1 file and the package file to the same directory. Run the sha1sum(1) command to verify the package file e.g. flowmon-1.1.0.tgz:
$ cat CHECKSUM.SHA1 | grep flowmon-1.1.0.tgz | sha1sum -c
If there are any errors, they will be reported.
/firmware - COMBO6(X) card firmware (*.mcs files)
/base - source code for software tools and drivers
/mk - build system (makefiles)
/sys_sw/drivers - kernel drivers
/sys_sw/hwtools - necessary hardware tools for COMBO6(X) card
/sys_sw/lib* - libraries necessary for other tools
/sys_sw/projects/flowmon - exporters, testing programs, configuration
and documentation for using FlowMon
/sys_sw/swtools/csxtool - tool for handle COMBO6(X) XML files
ERRATA - list of known bugs and their solutions
README - this file
RELNOTES - differences against previous release
The FlowMon hardware has been designed to work on any PC-AT compatible computer running GNU/Linux or other "Unix-like" operating system. It has been tested only on computers that use the x86 family of processors. The hardware requirements necessary to run FlowMon are:
The "sandwich" consisting of COMBO6(X) and an interface card (COMBO-4MTX, COMBO-4SFP or COMBO-4SFPRO) fits into one PCI slot.
| Mother card | Add-on card | Description |
|---|---|---|
| COMBO6 | a PCI (32/33) card with XILINX FPGA | |
| COMBO-4MTX | add-on card with four copper GE interfaces | |
| COMBO-4SFP | add-on card with four SFP cages for GE interfaces | |
| COMBO6X | a PCI-X (64/66) card with XILINX FPGAs | |
| COMBO-4SFPRO | add-on card with four SFP cages for GE interfaces |
| NOTE: | more information about COMBO6(X) cards you can get on http://www.liberouter.org/hardware.php |
|---|
FlowMon software works on GNU/Linux OS with 2.4 and 2.6 kernels. The NetFlow v5 and v9 protocols are supported. The software has been tested on computers running Red Hat Enterprise Linux, Ubuntu and Debian.
More information about FlowMon probe you can find at http://www.flowmon.org
After plugging COMBO6(X) card into your PCI slot, you should test connection between the card and your PC. We use lspci(8) utility for this purpose. lspci(8) is a utility for displaying information about all PCI buses in the system and all devices connected to them. For correct recognition of the COMBO6(X) card you need update PCI ID Database used by lspci(8) or download pciutils-2.2.2 (program collection containing lspci(8)) or later. If the lspci(8) output contains the following line your COMBO6(X) card is connected properly.
$ lspci -d 18ec: 03:01.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6X (rev 01) 04:02.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6 (rev 01)
| NOTE: | Numbers at the beginning of the line can be different. |
|---|
Compilation and installation process is covered by the build system located in the base/mk directory. For more details about required tools necessary to build package see base/mk/REQUIREMENTS file.
To build kernel drivers needed by FlowMon package you will need fully configured Linux 2.4 or 2.6 kernel source tree. More installation and compilation details about kernel drivers are available in the README file inside base/sys_sw/drivers/linux directory.
pkgtool(1) helps to build, install and uninstall the FlowMon distribution package.
$ tar -xzvf flowmon-XX.YY.ZZ.tgz $ cd flowmon-XX.YY.ZZ/base $ ./pkgtool --build
| NOTE: | Replace XX.YY.ZZ with the chosen package version number such as 1.1.0. |
|---|
Optionally you can define installation directory with the --prefix=path option. The --prefix option takes effect only during building package. If no prefix is set then /usr/local path is used.
$ ./pkgtool --build --prefix=/usr/local/flowmon
The tools and kernel drivers will be installed to the installation directory (by default /usr/local).
# ./pkgtool --install
If you are going to use udev mechanism to creating device files, you can use pkgtool with --udev option. This option cause copying file with COMBO6(X) card rules (combo6.udev.rules) to the /etc/udev/rules directory.
# ./pkgtool --install --udev
| NOTE: | --udev option takes effect only with --install option. |
|---|
There are necessary following post-install steps:
Set PATH variable to point to installed FlowMon's tools directory -> ${PREFIX}/bin
$ export PATH=$PATH:/usr/local/bin
Set /etc/ld.so.conf to point to the FlowMon's library directory -> ${PREFIX}/lib
Add /usr/local/lib (by default) entry to the /etc/ld.so.conf file.
# echo "/usr/local/lib" >> /etc/ld.so.conf # ldconfig
Set /etc/man.config file to point to the FlowMon's man pages -> ${PREFIX}/man
Add MANPATH /usr/local/man entry to the /etc/man.config file.
If you want to load FlowMon's kernel modules when the PC starts up, add following lines to the /etc/modules file.
For COMBO6 card:
combo6core combo6 szedata netflow-ph1
For COMBO6X card:
combo6core combo6x szedata szedatax-c6pcr
After this, you don't need to run flowmonlkm script any more.
| NOTE: | In some distribution (e.g. Red Hat Enterprise Linux 4) you have to use file /etc/rc.modules in place of /etc/modules file. This file contains commands to load kernel modules at boot time (it is actually script so it must be an executable file), so such file will look like this: |
|---|
#!/bin/sh /sbin/modprobe combo6core /sbin/modprobe combo6x /sbin/modprobe szedata /sbin/modprobe szedatax-c6pcr
The FlowMon probe driver needs device files in the /dev/combosix/ and /dev/szedata/ directory. To create them, run the flowdevices script in the packages directory base/sys_sw/drivers/linux
# ./flowdevices
To run FlowMon as a non-privileged user set r/w permissions on /dev/combosix/0 and /dev/szedata/0 devices.
You have to change the permissions of the device files so that the non-privileged user that will execute the FlowMon tools has read-write access. For example, you can create a new group, say combo-rw, add that user to this group and change the file group ownership and permissions as follows:
# chgrp combo-rw /dev/combosix/0 /dev/szedata/0 # chmod 664 /dev/combosix/0 /dev/szedata/0
Using recent 2.6 kernels it might be wanted to use the sysfs/udev mechanisms to create device files. For more details about udev see http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html
Example udev rules are placed in the package in sys_sw/drivers/linux/combo6.udev.rules file. If you didn't use option --udev during package installation, copy this file to /etc/udev/rules.d directory and modify it (change group or permissions). Following line will create COMBO6(X) device files required by the FlowMon probe:
``KERNEL=="combosix[0-9]*", NAME="combosix/%n", GROUP="combo-rw", MODE="0664"`` ``KERNEL=="szedata[0-9]*", NAME="szedata/%n", GROUP="combo-rw", MODE="0664"``
| NOTE: | Depending on your version of udevd you must adjust syntax of your udev rule file. Older versions of udevd require '=' character to match KERNEL key e.g. KERNEL="combosix[0-9]*". The newer one require '==' characters to match KERNEL key e.g. KERNEL=="combosix[0-9]*". |
|---|
If you have used some previous version of the FlowMon package, please follow all building and installing steps described above. Post-installation steps can be skipped if you performed these steps during previous installation (and an installation path was the same!). pkgtool(1) is able to detect your FlowMon Probe configuration file and it will ask you to decide if you wish to keep your own (but may be obsolete) configuration file or to overwrite them with our default (but up-to-date) configuration file.
It is not possible (for safety reasons) to automatically uninstall FlowMon package. pkgtool(1) will only show directories where the FlowMon tools are placed. By default the FlowMon package uses following files and directories:
/dev/combosix/ /dev/szedata/ /etc/liberouter/ /usr/local/bin/ /usr/local/etc/ /usr/local/include/ /usr/local/lib/ /usr/local/mcs/
| WARNING: | Some of the directories may be shared with other applications. Removing these directories may break your system. |
|---|
flowmon.conf is placed in the /etc/liberouter/ directory. It is used to store default probe settings and some other useful settings for the flowmon project scripts. flowmon.conf is connected to the scripts by the . (dot) shell command, so it is, actually, set of variable definitions and these variables are then used in our scripts. You can freely modify this file and redefine default values of the probe settings.
During installation of the next flowmon package version you will be inquired if you prefer to keep your own configuration file or to rewrite it with our up-to-date version. So you don't have to be worry to lose your changes. Up-to-date file is during installation process always copied to the $PREFIX/etc/liberouter (by default /usr/local/etc/liberouter) directory. So you can use this file as a backup copy.
As we wrote earlier, flowmon.conf has the form of the list of variables. Detailed information about these variables and their meanings you can get from flowmon.conf's man page:
$ man flowmon.conf
This procedure serves to start FlowMon probe - probe will not be automatically started again after rebooting PC. If you wish to start the FlowMon probe anytime you will reboot the PC, please see section FlowMon init.d script.
After building and installing package including post-install steps, all FlowMon's tools (included man pages) are available as any other system tools. There are two main scripts to start the probe - flowmon and flowmonlkm. Both scripts use /etc/liberouter/flowmon.conf configuration file.
For more details how to run flowmon* scripts use -h option or see flowmon's and flowmonlkm's man pages. Other FlowMon probe tools come with the manual pages. These manual pages can be viewed with the man(1) command e.g.
$ man flowmon_nf5
| NOTE: | Commands which begin with '#' must be run by root. To run commands as a non-privileged user '$' don't forget to set r/w permissions on /dev/combosix/0 and /dev/szedata/0 devices. |
|---|
flowmonlkm script is used for loading/removing FlowMon's kernel modules. There are two main options for this script. Option -l is used for loading FlowMon's kernel modules and option -r is used for removing these modules. Script uses lspci(8) utility to detect COMBO6(X) card.
When all kernel modules are loaded, csid(1) is able to detect type of plugged cards and provide more detailed information.
You should get following output on PC with installed
COMBO6 mother card and mtx add-on card:
$ csid combo6 mtx2 xcv2000
COMBO6X mother card and sfpro add-on card:
$ csid combo6x sfpro xc2vp20
| NOTE: | Chips can be different on some cards, so don't worry about differing last part of the csid's output |
|---|
flowmon is main start up script for the FlowMon probe. Script boots firmware files (according to FIRMWARE_PATH variable from the /etc/liberouter/flowmon.conf), setting up probe behavior and is able to start NetFlow protocol exporter. By default, the probe settings are set according to variables from /etc/liberouter/flowmon.conf configuration file. Optionally you can define your own settings by options given to the script. More detailed information you can get by -h option or from man page:
$ flowmon -h $ man flowmon
| NOTE: | Both scripts (flowmon as well as flowmonlkm) using lsof(8) tool to detect any process using COMBO6(X) device(s). To run scripts properly you have to install lsof(8) tool. lsof(8) is utility to get information about files opened by processes. |
|---|
The FlowMon probe logs important events to syslog using the user facility. To have these log messages saved to a file, you must configure the syslog daemon appropriately. In the case of the classical Unix syslogd daemon, add the following line to /etc/syslog.conf:
user.* /var/log/user.log
Regarding to your Linux distribution see syslogd(8) and logrotate(8) documentation how to adjust log file rotation.
# flowmonlkm -l $ flowmon -ec collector.liberouter.org:60000
To make sure that the firmware was loaded correctly you can run csid(1):
$ csid -s Board : combo6 Addon : mtx2 Chip : xcv2000 LAN ports: 4 Firmware : ok SW : 0xf1010002 HW : 0x00000006 Text : NETFLOW_1Gbps_Probe
HW number detected by csid -s reflects firmware version, e.g. HW : 0x00000006 reflects version NETFLOW_01_06.
To stop running exporter(s) and detach drivers, use flowmon script with the -s option:
$ flowmon -s
Then you can remove Flowmon's loaded kernel modules by the flowmonlkm script:
# flowmonlkm -r
Startup scripts are used to start some service (or some script) at the machine's boot time. Our sample startup script is used to automatically start up the FlowMon probe (loads kernel modules if necessary, boots firmware and set up the probe behavior) anytime your PC is starting up. Sample script is stored in the package directory structure in the base/sys_sw/projects/flowmon/doc directory as a flowmon.rc file. It is prepared for use in the SysV init system (and tested on the Red Hat Linux distribution). Script is commented so you can get a lot of information directly from the script.
Startup script uses flowmon.conf to start flow exporter(s).
copy flowmon.rc file to the /etc/rc.d/init.d/ directory and rename it to the flowmon
run chkconfig(8) and add flowmon as new service
# chkconfig --add flowmon
to verify, everything is right, you can try to find flowmon in the list of services
$ chkconfig --list | grep flowmon
reboot your system
To stop FlowMon (kill running exporters) go to the /etc/rc.d/init.d/ directory and use following command:
# ./flowmon stop
This command causes stopping FlowMon but the service will be started again during next reboot. To switch off the FlowMon service, in the current runlevel, use chkconfig(8) with off option after service name (flowmon in our case):
# chkconfig flowmon off
Last possibility is to remove FlowMon service from chkconfig management:
# chkconfig --del flowmon
| NOTE: | More detailed information you can find in the chkconfig's man page. |
|---|
To show current status of the FlowMon probe use option status. It displays all running FlowMon's exporter(s).
# ./flowmon status
To read data from HW and send them to collector you can run several instances of FlowMon exporters. Before running exporter you must start FlowMon probe (load kernel modules and run flowmon script).
$ flowmon_nf5 -d collector.liberouter.org:60000 $ flowmon_nf9 -d collector.liberouter.org:60001
The exporter program will start sending flow records to the configured collector. To test that it is really the case, you can run a packet sniffer such as tcpdump(1) or ethereal(1), for example
# tcpdump -i <interface> 'udp dst port <collector_port>' # tcpdump -i eth0 'udp dst port 60000'
You should see the traffic sent by the flow exporter to the configured collector.
There are several implementations of NetFlow v5, v9 collectors. Some of them handle NetFlow datagrams in different manner. We recommend using NfSen - Netflow Sensor - http://nfsen.sourceforge.net/ collector.
Repeater duplicates data from PORT0 to PORT1 and vice versa. PORT3 and PORT4 are not utilized at all. Input stream is processed by IBUF. In each IBUF Input Sampling rate can be set. Repeater is working properly but when the computer is turned off, repeater does not work. During reloading or resetting card, it takes about 3 seconds to start repeating again. Also monitoring starts 3 seconds later after reset or reload.
+--------+ +------+ +-----+ +---------+ +---------+
| PORT 0 |->---+->-| IBUF |-->| HFE |-->| UH_FIFO |-->| UHDRV |
| 1 Gb/s |-<-+ | +------+ +-----+ +---------+ +---------+
+--------+ | | |
| | v
+---+-+----+ +---------+
| REPEATER | | MONITOR |
+---+-+----+ +---------+
| | |
+--------+ | | v
| PORT 1 |->-+ | +---------+
| 1 Gb/s |-<---+ | SW |
+--------+ +---------+
Fig. 1 - FlowMon probe block schema
If you want to check FlowMon features you can simply mirror traffic from your router to the FlowMon probe.
+------------+
| Internet |
+------------+
|
|
|
+----------+ 1 Gb/s +---------------+ +-----------+
| Router |-->------------->--| FlowMon Probe |-->----->--| Collector |
+----------+ mirror port 0 +---------------+ LAN +-----------+
| port
|
|
+------------+
| Intranet |
+------------+
Fig. 2 - Simple FlowMon probe connection schema
where
- mirror port : the copy of the traffic passing through the router
- port 0 : port 0 of MTX or SFP interface card (the closest one to the PCI slot)
- LAN : local network connection to collector
You can also use the repeater functionality of the probe to connect it before your border router. This way you can monitor all the incoming or outgoing traffic.
+------------+
| Internet |
+------------+
|
|
| port 0
+---------------+ +-----------+
| FlowMon Probe |-->----->--| Collector |
+---------------+ LAN +-----------+
| port 1
|
|
+------------+
| Router |
+------------+
|
|
|
+------------+
| Intranet |
+------------+
Fig. 3 - FlowMon Probe inserted in a line
where
- port 0 : port 0 of MTX or SFP interface card (the closest one to the PCI slot)
- port 1 : port 1 of MTX or SFP interface card
- LAN : local network connection to collector
The firmware for COMBO-4MTX and COMBO-4SFP interface card should be able to process 230 000 packets/s or up to 750 Mb/s. If you are monitoring higher traffic you must set IBUF sampling.
$ flowmon -S 0:2 -T 0:0 -ec collector.liberouter.org:60000
The probe interface can handle properly only 1 GE.
Some unusual way of extracting the package may break down symbolic links located in firmware/index/NETFLOW_1Gbps_Probe/. To repair links simply run firmware/make_index script.
Being designed for high-rate flow troughput the FlowMon software hasn't got any "packet-level" exporting timeouts (it simply waits till its 1.5 kB buffer gets filled with records). This may cause exporting delays especially on filtered low-rate traffic sources.
The errata document contains additional information on how to handle known FlowMon probe problems. It is important to consult this document to learn about any post-release discoveries or problems that may already have been found and fixed.
For any questions or general technical support issues, please send mail to the netflow [at] liberouter.org general mailing list.
Suggestions, bug reports and contributions of code are always valued. Please do not hesitate to report any problems you may find. If you encounter any suspicious behavior of your FlowMon probe (stops monitoring, reports non existing flows, runs too slow, reports only few flows, ... ) please contact us.
Before submitting a problem, be sure you are running the latest FlowMon package version. Please send the following information to us:
This will help us track the problem and resolve it. Bug reports with attached fixes are of course even more welcome.
Check our web pages for information about other projects, e.g. NIFIC (network interface card with packet filtering and forwarding) or intrusion detection system.
| [1] | Celeda, P.; Kovacik, M; Konir, T.; Krmicek, V.; Springl, P.; Zadnik, M. FlowMon probe, Cesnet technical report 31/2006, available online. |
| [2] | Celeda, P.; Kovacik, M.; Krejci, R.; Kysela, J.; Springl, P. Software for NetFlow Monitoring Adapter, Cesnet technical report 33/2005, available online. |
| [3] | Zadnik, M.; Lhotka, L. Hardware-Accelerated NetFlow Probe, Cesnet technical report 32/2005, available online. |
| [4] | Cisco Systems Cisco Systems NetFlow Services Export Version 9, available online. |
| [5] | Cisco Systems NetFlow Export Datagram Formats, available online. |