Network Interface Card over NetCOPE Handbook

The NIC Card is a basic network device for packet reception and transmission based on COMBO6X technology. It is able to receive and transmit packets on various speeds (1Gbps, 100Mbps and 10Mbps). Additionally this card supports precise timestamps, that can be very useful for instance for measuring of latencies in your network topology. This information can be later used for improving your topology or network device faults detection.

This NIC Card has been developed on NetCOPE platform as its simplest application. The NetCOPE platform supports rapid development of network application on FPGA accelerated cards by providing low-level abstraction layer for each card. This layer encapsulates network, system bus controllers or DMA transfers and thus speeds up application development. One of the main targets of NetCOPE on each card is to achieve maximal data throughput between user acceleration core on FPGA and software applications. In case of the NIC Card, data are transferred directly from Input blocks to user software applications without any modifications in user application core on FPGA.

SW Parameters of NIC Card

NetCOPE platform provides these interfaces to software:

1. Standard network interface - NIC over NetCOPE acts in system as ordinary network interface. Received packets are passed from driver to operating system kernel and further through TCP stack to target application. Alternatively PCAP library, part of operating system, can be used to access packets in raw format avoiding passage through TCP stack (typical for monitoring applications as Tcpdump, Wireshark, Snort etc.). Advantage of this interface is its universality and possibility of using existing applications. Disadvantage is low throughput caused by passage through operating system kernel which requires disassembling packets into segments without exploiting advantages of aggregation.

2. Application specific interface - NetCOPE card provides general interface for block transfers to applications. Communication takes place directly between driver and userspace application without passage through TCP stack. Format of passed data is not restricted by operating system kernel and therefore this interface can benefit from using aggregated packets with control information, timestamps etc. appended. Except packet data various data (depending on accelerator purpose) such as netflow records, information extracted form packet headers, etc. can be transferred. NIC over NetCOPE uses this interface to transfer aggregated packets. In addition PCAP library adjusted to work within this interface, communicating directly to driver, avoiding passage through operating system kernel, has been created and is provided. Similar approach use companies as Endace, Napatech, etc.. Main advantage of this approach is high throughput using aggregation of smaller packets reducing DMA transfers overhead.

Figure 1.1. HW SW Communication Models

NIC package is available via WWW download page: http://www.liberouter.org/clients

To verify that the downloaded files are genuine and complete SHA1 digests are available. Download both the CHECKSUM.SHA1 file and the package file to the same directory. Run the sha1sum(1) command to verify the package file e.g. nic-2.0.0.tgz:

$ cat CHECKSUM.SHA1 | grep nic-2.0.0.tgz | sha1sum -c
      

If there are any errors in the package file integrity, they will be reported.

	/firmware      - COMBO6X card firmware (*.mcs files)
	/base          - source code for software tools and drivers
	  /mk                        - build system (makefiles)
	  /sys_sw/drivers            - kernel drivers
	  /sys_sw/hwtools            - necessary hardware tools for COMBO6X card
	  /sys_sw/lib*               - libraries necessary for other tools
    /sys_sw/projects/nic_netcope - configuration files and scripts for using NIC
    /sys_sw/swtools/csxtool    - tool for handle COMBO6X XML files
	/doc           - NIC project documentation - NIC Handbook
	README         - short manual how to build, install and use NIC Card
	RELNOTES       - differences against previous release

This package can be used with the COMBO6X mother card and COMBO-4SFPRO add-on card with four SFP cages for GE interfaces.

Figure 2.1. COMBO6X Card

NIC Card software works on GNU/Linux OS with 2.4 and 2.6 kernels.

After plugging COMBO6X card into your PCI slot, you should test connection between the card and your PC. We use lspci(8) utility for this purpose. lspci(8) is a utility for displaying information about all PCI buses in the system and all devices connected to them. For correct recognition of the COMBO6X card you need update PCI ID Database used by lspci(8) or download pciutils-2.2.2 (program collection containing lspci(8) or later. If the lspci(8) output contains the following line your COMBO6X card is connected properly.

	$ lspci -d 18ec:
	03:01.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6X (rev 01)
	

NIC Card Port Numbers

The general rule for numbering card ports is that the ports closer to the motherboard (PCI slot) have lower numbers, e.g. the closest port has number 0, the next one number 1 etc. See figures bellow for examples:

Figure 2.2. NIC Card COMBO-4SFPRO Card Port Numbers

pkgtool(1) helps to build, install and uninstall the NIC distribution package.

	$ tar -xzvf nic-XX.YY.ZZ.tgz
	$ cd nic-XX.YY.ZZ/base
	$ ./pkgtool --build
	

If this is the first Liberouter package you have ever installed, you can define installation directory with the --prefix=path option. But remember that this installation path MUST not exists (e.g. /usr/local is invalid installation path on most systems because this directory exists). This restriction is due to new installation framework which enables easy package uninstall or package switching. More information about these features can be found in the liberouterpkg section.

The --prefix option takes effect only during building package. If no prefix is set then /usr/local/liberouter path is used.

	$ ./pkgtool --build --prefix=/usr/local/nic
    

If you have previously installed some Liberouter package (flowmon, nic, nific, ids with liberouterpkg mechanism), the installation path is detected automatically as path used for the first installed package.

Remember that installation path given as --prefix parameter will contain next subdirectories for binaries, libraries, man pages, etc. These directories can be affected by future uninstalling or package switching so it could be used as installation directory only for Liberouter packages.

The tools and kernel drivers will be installed to the installation directory (by default /usr/local/liberouter)).

	# ./pkgtool --install
	

If you are going to use udev mechanism to creating device files, you can use pkgtool with --udev option. This option cause copying file with COMBO6X card rules (combo6.udev.rules) to the /etc/udev/rules directory.

	# ./pkgtool --install --udev
	

There are necessary following post-install steps:

All NIC tools come with its manual pages so for information about any tool (included configuration files) you can see these man pages by man(1) program, e.g.

        $ man nic
        $ man nic.conf
        

The nic-2.0.0 is the Liberouter package using liberouterpkg tool enabling package switching and fully package uninstalling. If you have installed any previous Liberouter package without liberouterpkg, please remove it completely (including all libraries and drivers) to ensure proper behavior of the installed package.

More information about these features can be found in the liberouterpkg section.

If you are installing new version of previously installed package (e.g. you have installed nic-2.0.0 and now you are installing nic-2.0.1 package) you will be asked by pkgtool(1) to decide if you wish to keep your own (but may be obsolete) configuration files or to overwrite them with our default (but up-to-date) configuration files.

liberouterpkg script is new tool covering new Liberouter package installation framework which enables safe and easy package uninstalling or simple package switching (and using different project packages on the same PC). To display all available functions of the liberouterpkg script use --help option

    $ liberouterpkg --help
    

liberouterpkg uses /etc/liberouter/packages.list configuration file that stores information about installed packages.

      # liberouterpkg --list
      # Installed packages:
      nic-2.0.0
      

      # liberouterpkg
      nic-2.0.0
      

      # liberouterpkg --uninstall=nic-2.0.0
      

      # liberouterpkg --switch=flowmon-1.3.0
      

These variables are only used when aggregation is enabled by setting AGREGATOR="YES".

These variables are ignored when aggregation is enabled by setting AGREGATOR="YES".

You need to specify four network interfaces (N=[0-3]).

For loading/removing NIC Card kernel modules the niclkm(1) script is used. This script must be executed with root privileges. There are two main options for this script. Option -l is used for loading NIC Card kernel modules. Modules are loaded appropriately to the selected mode (no aggregation, aggregation, ptm timestamps) in the nic.conf. Option -r is used to remove any modules previously loaded. Script supports only combo6x card. Detection of the card is performed by the lspci(8) utility.

Combosix device "/dev/combosix/0" doesn't exist.
Please create all necessary devices.
    

When all kernel modules are loaded, csid (1) is able to detect type of plugged cards and provide more detailed information. You should get following output on PC with installed combo6x mother card and sfpro add-on card:

$ csid
combo6x sfpro xc2vp20

nic(1) is the main start up script for NIC design. Script boots firmware files (according to FIRMWARE_PATH variable from the /etc/liberouter/nic.conf) and configures combo card interfaces. To simply set up NIC, run nic script without any parameters:

$ nic

To shut down combo card interfaces and detach drivers (actually stop NIC), run the nic script with -s option:

$ nic -s

-----------------------------------------------------
Attaching driver
-----------------------------------------------------
csboot: unable to attach driver: Cannot allocate memory
/usr/local/liberouter/bin/nic: Failed to attach driver!

More information about using nic script can be displayed by -h option.

Booted firmware can be checked by csid (1):

$ csid -s
Board    : combo6x
Addon    : sfpro
Chip     : xc2vp20
LAN ports: 4
Firmware : ok
SW       : 0x41c10200
HW       : 0x00030000
Text     : NIC_NetCOPE
PCI brver: c610.05.09 (2007/11/09 14:30) NetCOPE Bridge
PCI PPC  : [nic Nov  9 2007 21:02:59] 0xdeadbaff -booted- -running-

To use NIC Card in a mode with standard network interfaces, which are configurable by e.g. ifconfig, you need to edit nic.conf configuration file. This file is described in Chapter 3, Configuration of the NIC Card.

After saving your settings, run these commands:

# niclkm -l
$ nic

After the execution of those scripts NIC Card is started and ready to use. There are four new network interfaces in the system. Numbers in the interface names are set according to number of card plugged in the PC (the first digit) and according to number of interface (second digit).

You can check that NIC Card is set up correctly by setting desired IP address to NIC interfaces (via nic.conf or ifconfig) and then ping (8) it from PC connected to some NIC's port. NIC should answer to echo request by echo reply.

To provide high throughput interface to access data from hardware, SZEDATA interface has been created in form of kernel modules szedata and szedatax_c6pcr. SZEDATA stands for straight zero-copy data interface API. More described in introduction section.

Use of NIC Card with SZEDATA interface is called Aggregation Mode.

To use NIC Card in aggregation mode you need to edit nic.conf configuration file. This file is described in Chapter 3, Configuration of the NIC Card.

After saving your settings, run these commands:

# niclkm -l
$ nic

After the execution of those scripts NIC Card is started and ready to use. There are eight new szedata interfaces in /dev/szedata/.

 7  6  5  4  3  2  1  0

 00 a2 a1 a0 00 03 00 3c
 07 06 05 04 03 02 01 00
 0f 0e 0d 0c 0b 0a 09 08
 17 16 15 14 13 12 11 10
 1f 1e 1d 1c 1b 1a 19 18
 27 26 25 24 23 22 21 20
 2f 2e 2d 2c 2b 2a 29 28
 37 36 35 34 33 32 31 30
 00 00 00 00 3b 3a 39 38

Well known libpcap library has been modified to support szedata interface as regular NIC. It allows to use widely used applications such as tcpdump, wireshark, nprobe or fprobe with benefits of Liberouter NIC Card or create new applications using PCAP API.

Detailed howto and examples can be found at tcpdump webpages (www.tcpdump.org)

To use libpcap-sze library, NIC Card must be run in Aggregation Mode, please see the section called “Aggregation Mode and Timestamps”.

Startup scripts are used to start some service (or some script) at the machine's boot time. Our sample startup script is used to automatically start up the NIC (loads kernel modules, boots firmware and set up network interfaces) anytime your PC is starting up. Sample script is stored in the package directory structure in the base/sys_sw/projects/nic_netcope/doc directory as a nic.rc file. It is prepared for use in the SysV init system (tested on the Red Hat Linux distribution). Script is commented so you can get a lot of information directly from the script.

Provided sample startup script uses the nic.conf file.

# ./nic stop

# chkconfig nic off

# chkconfig --del nic

# ./nic status

For correct functionality changing the speed requires two steps. First the transceiver speed must be changed using the tool phyterctl(1). Next the IBUF component speed needs to be updated corresponding to the transceiver speed using the tool ibufctl(1).

phyterctl(1) is a tool used to display and change configuration of 4 interfaces available on COMBO-4SFPRO add-on cards. The tool displays information about link status, resolved speed or duplex mode on link. phyterctl(1) is also able to change the advertised speed and duplex mode and provides r/w access to internal registers of the physical layer IC.

ibufctl(1) is used to display and change configuration of IBUF components in COMBO6X designs.

Example of changing the speed on interface 0 to 100Mbps.

	$ phyterctl -c speed 100 -i0 ... set transceiver speed to 100Mbps on interface 0
	$ ibufctl -s100 -i0 ... set IBUF speed to 100Mbps on interface 0

Example of phyterctl(1) listing with GBIC EEPROM information:

	$ phyterctl -c gbic
	Settings for card 0 (device /dev/combosix/0):
	------------------------------ Interface 0 ---
	Transceiver      FINISAR CORP.
	Model		 FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	------------------------------ Interface 1 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	------------------------------ Interface 2 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Link status      Down
	------------------------------ Interface 3 ---
	Transceiver      FINISAR CORP.
	Model            FCMJ-8521-3
	Phyter vendor    MARVELL
	Phyter model     88E1111 Gigabit PHY
	Speed            1000 Mb/s
	Mode             Full-duplex
	Link status      Up
	

More information can be found in the phyterctl(1) and ibufctl(1) man pages or in the README files placed in the base/sys_sw/hwtools/phyter/ and base/sys_sw/hwtools/ibufctl/ directories.

Crossbar component provides option that every PORT (RX part) and INPUT can be assigned to zero or more PORTS (TX part) or OUTPUTS. Vice versa, every PORT (TX part) and OUTPUT must have assigned exactly one of PORTS (RX part) or INPUTS.

crossbarctl(1) is a command line tool used to display current configuration and to connect ports, outputs and inputs.

Figure 4.1. Crossbar default state

$ crossbarctl -l
p0 <== obuf0
p1 <== obuf1
p2 <== obuf2
p3 <== obuf3
--------
ibuf0 <== p0
ibuf1 <== p1
ibuf2 <== p2
ibuf3 <== p3

Example of connecting OUTPUT2 with INPUT3.

$ crossbarctl -s obuf2 -t ibuf3 

Example of connecting PORT1 with INPUT3.

$ crossbarctl -s p1 -t ibuf2

More information can be found in the crossbarctl(1) man page.

Information about created devices /dev/combosix/* and /dev/szedata/* are held after creation in /proc directory.

$ cat /proc/driver/combo6/version 
Combo6 Driver Version 0.3.57.
Compiled on Feb 14 2008 09:11:04 for kernel 2.6.9-11.EL.

$ cat /proc/driver/combo6/card0/id 
Board    : combo6x
Addon    : sfpro
Chip     : xc2vp20
LAN ports: 4
Firmware : none
SW       : 0x00000000
HW       : 0x00000000
Text     : unknown
PCI brver: c610.05.0a (2007/11/22 19:55) NetCOPE Bridge
PCI PPC  : [] 0x0 -not loaded- -inactive-

Device [combo6x] c6pcreth
(0xabcdef02-0xabcdef02) {}: inactive
Device [combo6x] c6pcreth
(0x4f1c0000-0x4f1c0000) {NIFIC_1Gbps}: inactive
Device [combo6x] c6pcreth
(0x4f1c0000-0x4f1c0000) {Nific4_1Gb_COMBO6X}: inactive
Device [combo6x] c6pcreth
(0x4f1c0101-0x4f1c01ff) {Liberouter (4-NIC filtration)}: inactive
Device [combo6x] c6pcreth
(0x4f1c0100-0x4f1c01ff) {NIFIC_NetCOPE}: inactive
Device [combo6x] netcope-pcreth
(0x41c10200-0x41c102ff) {NIC_NetCOPE}: inactive
Device [combo6x] netflow-ph1-pcr
(0xf1010001-0xf10102ff) {NETFLOW_1Gbps_Probe}: inactive
Device [combo6x] scampi-ph2-pcr
(0x5ca20001-0x5ca20001) {SCAMPI 10Gbps Monitoring adapter}:
inactive
Device [combo6x] scampi-ids1-pcr
(0x1d510001-0x1d510001) {IDS_1Gbps_Traffic_Scanner}: inactive
Device [combo6x] scampi-ids2-pcr
(0x1d510200-0x1d510200) {IDS_1Gbps_Traffic_Scanner}:
inactive

$ cat /proc/driver/szedata/version
SZEData Driver Version 0.3.57.
Compiled on Feb 14 2008 09:11:14 for kernel 2.6.9-11.EL.

cat /proc/driver/szedata/device0/stats 
General:
Alloc blocks:         65536		/* number of allocated blocks */
Alloc bsize:          4096		/* allocated block size */
Alloc mmap:           268435456		/* allocated memory - product of two numbers above */
Alloc failed:         0			/* failed allocations */
Alloc over:           0			
Active apps:          00000001		/* number of active applications using szedata interface */
Running apps:         00000001		/* number of running applications using szedata interface */
Application #0:				/* INFORMATION about running applications */
Subs. interfaces:     0000000f		/* subscribed interfaces mask - 0000000f means interfaces 0,1,2,3 */
Waiting packets:      0			
Waiting peak pkts:    1			
Poll threshold:       1			 
Total locks:          497
Total unlocks:        497
Application #1:
Subs. interfaces:     00000000
Waiting packets:      0
Waiting peak pkts:    0
Poll threshold:       0
Total locks:          0
Total unlocks:        0
Application #2:
Subs. interfaces:     00000000
Waiting packets:      0
Waiting peak pkts:    0
Poll threshold:       0
Total locks:          0
Total unlocks:        0
Application #3:
Subs. interfaces:
00000000
Waiting packets:      0
Waiting peak pkts:    0
Poll threshold:       0
Total locks:          0
Total unlocks:        0

Suggestions, bug reports and contributions of code are always valued. Please do not hesitate to report any problems you may find. If you encounter any suspicious behavior of your NIC Card please contact us.

Before submitting a problem, be sure you are running the latest NIC Card package version. Please send the following information to us:

This will help us track the problem and resolve it. Bug reports with attached fixes are of course even more welcome.

Check our web pages for information about other projects, e.g. FlowMon (flexible flow monitoring system) or intrusion detection system.

http://www.liberouter.org/