Cesnet Liberouter
  • Projects
  • Liberouter
  • Scampi
  • FlowMon
  • NIC
  • NIFIC
  • IDS
  • NetCOPE
  • VHDL design
  • System software
  • Testing
  • Formal verification
  • Netopeer
  • Documents
  • Our hardware
  • Card Availability
  • Our partners
 

NIC - 4-port Network Interface Card

Date: April 2008
Version: 2.0.0
Author: The Liberouter Project Team
Organization: CESNET, z.s.p.o.

Contents

  • 1   Purpose of this README
  • 2   Introduction
    • 2.1   Obtaining NIC package
    • 2.2   Package structure
  • 3   Installation instructions
    • 3.1   Host computer
    • 3.2   Supported hardware
    • 3.3   Supported software
    • 3.4   Installing COMBO6X cards
    • 3.5   Building package
    • 3.6   Installing package
    • 3.7   Upgrading package
  • 4   Using NIC Card
    • 4.1   Loading kernel modules
    • 4.2   Setting up NIC Card
      • 4.2.1   Loading kernel modules and running the NIC Card
      • 4.2.2   Stopping the NIC Card and removing kernel modules
  • 5   Contacting the NIC team
    • 5.1   Mailing lists

1   Purpose of this README

This README document is used as short manual to build, install and simple use (functionality test). More detailed information about using NIC and its configuration is available in the NIC Handbook.

Please do not use this README only as a step-by-step manual and read all of README first to familiarize with all available features.

2   Introduction

The NIC Card is a basic network device for packet reception and transmission based on COMBO6X technology. It is able to receive and transmit packets on various speeds (1Gbps, 100Mbps and 10Mbps). This card also supports precise timestamps, that can be very useful for instance for measuring of latencies in your network topology. This information can be later used for improving your topology or network device faults detection.

This NIC Card has been developed on NetCOPE platform as it's simplest application. The NetCOPE platform supports rapid development of network application on FPGA accelerated cards by providing low-level abstraction layer for each card. This layer encapsulates network, system bus controllers or DMA transfers and speeds up application development. One of the main targets of NetCOPE on each card is to achieve maximal data throughput between user acceleration core on FPGA and software applications. In case of the NIC Card, data are transfered directly from Input blocks to user software applications without any modifications in application core on FPGA.

You can also check latest news at our web pages:

http://www.liberouter.org/
http://www.liberouter.org/nic/index.php

2.1   Obtaining NIC package

NIC package is available via WWW download page:

http://www.liberouter.org/clients

To verify that the downloaded files are genuine and complete SHA1 digests are available. Download both the CHECKSUM.SHA1 file and the package file to the same directory. Run the sha1sum(1) command to verify the package file e.g. nic-2.0.0.tgz:

$ cat CHECKSUM.SHA1 | grep nic-2.0.0.tgz | sha1sum -c

If there are any errors, they will be reported.

2.2   Package structure

The NIC package contains firmware and software to run NIC on COMBO6X cards. The package also contains NIC installation system, README, RELNOTES and NIC Handbook.

/base       - source codes for software tools and drivers
  /mk                          - build system
  /sys_sw/drivers/linux        - Linux kernel drivers
  /sys_sw/hwtools              - necessary hardware tools for COMBO6(x) card
  /sys_sw/lib*                 - libraries necessary for other tools
  /sys_sw/projects/nic_netcope - configuration files and scripts for using NIC
  /sys_sw/swtools/csxtool             - tool for handle COMBO6 XML files
/doc        - NIC project documentation - NIC (over NetCOPE) Handbook
/firmware   - card firmware (*.mcs files)
README      - this file
RELNOTES    - differences against previous release

3   Installation instructions

3.1   Host computer

The NIC hardware has been designed to work on any PC-AT compatible computer running GNU/Linux or other "Unix-like" operating system. It has been tested only on computers that use the x86 family of processors. The hardware requirements necessary to run NIC are:

  • Intel Pentium or equivalent x86 based PC
  • COMBO6X - 64-bit/66 MHz PCI bus

The "sandwich" consisting of COMBO6X and an interface card COMBO-4SFPRO fits into one PCI slot.

3.2   Supported hardware

Mother card Add-on card Description
COMBO6X   a PCI-X (64/66) card with XILINX FPGAs
COMBO-4SFPRO add-on card with four SFP cages for GE interfaces
NOTE:more information about Combo cards you can get on http://www.liberouter.org/hardware.php

3.3   Supported software

NIC software works on GNU/Linux OS with 2.4 and 2.6 kernels. The software has been tested on computers running Red Hat Enterprise Linux, Ubuntu and Debian.

More information about NIC project you can find at http://www.liberouter.org/nic/index.php

3.4   Installing COMBO6X cards

After plugging Combo card into your PCI slot, you should test connection between the card and your PC. We use lspci(8) utility for this purpose. lspci(8) is a utility for displaying information about all PCI buses in the system and all devices connected to them. For correct recognition of the Combo card you need update PCI ID Database used by lspci(8) or download pciutils-2.2.2 (program collection containing lspci(8)) or later. If the lspci(8) output contains the following line (according to the type of installed Combo card), your Combo card is connected properly.

$ lspci -d 18ec:
03:01.0 Ethernet controller: Cesnet, z.s.p.o. COMBO6X (rev 01)
NOTE:Numbers at the beginning of the line can be different.

3.5   Building package

Compilation and installation process is covered by the build system located in the base/mk directory. For more details about required tools necessary to build package see base/mk/REQUIREMENTS file.

To build kernel drivers needed by NIC package you will need fully configured Linux 2.4 or 2.6 kernel source tree. More installation and compilation details about kernel drivers are available in the README file inside base/sys_sw/drivers/linux directory.

pkgtool(1) is used to build and install the NIC distribution package.

$ tar -xzvf nic-XX.YY.ZZ.tgz
$ cd nic-XX.YY.ZZ/base
$ ./pkgtool --build
NOTE:Replace XX.YY.ZZ with the chosen package version number such as 2.0.0.

If this is the first Liberouter package you have ever installed, you can define installation directory with the --prefix=path option. But remember that this installation path MUST not exists (e.g. /usr/local is invalid installation path on most systems because this directory exists). This restriction is due to new installation framework which enables easy package uninstall or package switching. More information about these features can be found in the NIC Handbook in the liberouterpkg section.

The --prefix option takes effect only during building package. If no prefix is set then /usr/local/liberouter path is used.

$ ./pkgtool --build --prefix=/usr/local/nic

If you have previously installed some Liberouter package (nific, nic, ids, flowmon with liberouterpkg mechanism), the installation path is detected automatically as path used for the first installed package.

Remember that installation path given as --prefix parameter will contain next subdirectories for binaries, libraries, man pages, etc. These directories can be affected by future uninstalling or package switching so it could be used as installation directory only for Liberouter packages.

3.6   Installing package

The tools and kernel drivers will be installed to the installation directory (by default /usr/local/liberouter).

# ./pkgtool --install

If you are going to use udev mechanism to creating device files, you can use pkgtool with --udev option. This option cause copying file with COMBO6(X) card rules (combo6.udev.rules) to the /etc/udev/rules directory.

# ./pkgtool --install --udev
NOTE:--udev option takes effect only with --install option.

There are necessary following post-install steps which have to be done. Post-installation steps can be skipped (except last one) if you performed these steps during previous installation.

  1. Export Liberouter environment variables generated by pkgtool(1). These variables were printed as last output of pkgtool --install command and they are in the following form

    VARIABLE=value

    You can simple export them:

    $ export CS_XML_INDEX_PATH=/usr/local/liberouter/mcs/index
    

    or you can add them as following line(s) into the /etc/profile configuration file for permanent export

    export CS_XML_INDEX_PATH=/usr/local/liberouter/mcs/index
    

    Liberouter environment variables are prepared only if they are needed. So if pkgtool(1) doesn't print any "Liberouter environment variables:" you doesn't need export any variable.

  2. Set PATH variable to point to installed NIC tools directory -> ${PREFIX}/bin

    $ export PATH=/usr/local/liberouter/bin:$PATH
    

    For permanent PATH change add following line into the /etc/profile configuration file

    export PATH=/usr/local/liberouter/bin:$PATH
    
  3. Configure dynamic linker run time bindings by setting /etc/ld.so.conf to point to the NIC library directory -> ${PREFIX}/lib

    Add /usr/local/liberouter/lib (by default) entry to the /etc/ld.so.conf file.

    # echo "/usr/local/liberouter/lib" >> /etc/ld.so.conf
    # ldconfig
    
  4. Set /etc/man.config file to point to the NIC man pages -> ${PREFIX}/man

    Add MANPATH /usr/local/liberouter/man entry to the /etc/man.config file (name of this file can little differ on different Linux distribution.

  5. If you want to load NIC's kernel modules on Linux systems when the PC starts up, add following lines to the /etc/modules file.

    For COMBO6X card when using szedata interface:

    combo6core
    combo6x
    szedata
    szedatax_c6pcr blocks=65536 block_size=4096
    

    For COMBO6X card when using standard network interface:

    combo6core
    combo6x
    combo6x-c6pcreth
    

    After this, you don't need to run niclkm script any more.

    NOTE:

    In some distribution (e.g. Red Hat Enterprise Linux 4) you have to use file /etc/rc.modules in place of /etc/modules file. This file contains commands to load kernel modules at boot time (it is actually script so it must be an executable file), so such file will look like this:

    For COMBO6X card when using szedata interface:

    #!/bin/sh
    /sbin/modprobe combo6core
    /sbin/modprobe combo6x
    /sbin/modprobe szedata
    /sbin/modprobe szedatax-c6pcr
    

    For COMBO6X card when using standard network interface:

    #!/bin/sh
    /sbin/modprobe combo6core
    /sbin/modprobe combo6x
    /sbin/modprobe combo6x-c6pcreth
    
  6. The NIC driver needs device files in the /dev/combosix/ and /dev/szedata/ directory. To create them, run the combo6devices and szedatadevices scripts in the packages directory base/sys_sw/drivers/linux

    # ./combo6devices
    # ./szedatadevices
    

    To run NIC as a non-privileged user set r/w permissions on /dev/combosix/0 and /dev/szedata/0 devices.

    You have to change the permissions of the device files so that the non-privileged user that will execute the NIC tools has read-write access. For example, you can create a new group, say combo-rw, add that user to this group and change the file group ownership and permissions as follows:

    # chgrp combo-rw /dev/combosix/0 /dev/szedata/0
    # chmod 664 /dev/combosix/0 /dev/szedata/0
    

    Using recent 2.6 kernels it might be wanted to use the sysfs/udev mechanisms to create device files. For more details about udev see http://www.kernel.org/pub/linux/utils/kernel/hotplug/udev.html

    Example udev rules are placed in the package in sys_sw/drivers/linux/combo6.udev.rules file. If you didn't use option --udev during package installation, copy this file to /etc/udev/rules.d directory and modify it (change group or permissions). Following line will create COMBO6X device files required by the NIC card:

    ``KERNEL=="combosix[0-9]*", NAME="combosix/%n", GROUP="combo-rw", MODE="0664"``
    ``KERNEL=="szedata[0-9]*", NAME="szedata/%n", GROUP="combo-rw", MODE="0664"``
    
    NOTE:

    Depending on your version of udevd you must adjust syntax of your udev rule file. Older versions of udevd require '=' character to match KERNEL key e.g. KERNEL="combosix[0-9]*". The newer one require '==' characters to match KERNEL key e.g. KERNEL=="combosix[0-9]*".

  7. Please copy (and rewrite any previous version) liberouterpkg script, located in the package base directory into your standard binary directory (e.g. /usr/local/bin/). This enables using liberouterpkg script as standard system tool:

    $ liberouterpkg
    
NOTE:Path where the liberouterpkg script is placed must be different from used package installation path.

3.7   Upgrading package

This is the first Liberouter package using liberouterpkg tool enabling package switching and fully package uninstalling. If you have installed any previous Liberouter package without liberouterpkg, please remove it completely (including all libraries and drivers) to ensure proper behaviour of the installed package.

More information about these features can be found in the NIC Handbook in the liberouterpkg section.

If you are installing new version of previously installed package (e.g. you have installed nic-2.0.0 and now you are installing nic-2.0.1 package) you will be asked by pkgtool(1) to decide if you wish to keep your own (but may be obsolete) configuration files or to overwrite them with our default (but up-to-date) configuration files.

4   Using NIC Card

This procedure serves to start NIC Card - it will not be automatically started again after rebooting the PC. If you wish to start the NIC Card anytime you will reboot the PC, please see the NIC Handbook, section init.d script.

After building and installing package including post-install steps, all NIC tools (included man pages) are available as any other system tools. There are two main scripts to start the card - nic and niclkm. Both scripts use /etc/liberouter/nic.conf configuration file.

For more details how to run nic* scripts use -h option or see nic and niclkm man pages. Other NIC Card tools come with the manual pages. These manual pages can be viewed with the man(1) command e.g.

$ man niclkm
NOTE:Commands which begin with '#' must be run by root. To run commands as a non-privileged user '$' don't forget to set r/w permissions on /dev/combosix/0 and /dev/szedata/0 devices.

4.1   Loading kernel modules

niclkm script is used for loading/removing NIC kernel modules. There are two main options for this script. Option -l is used for loading NIC kernel modules and option -r is used for removing these modules. Script uses lspci(8) utility to detect COMBO6(X) card.

When all kernel modules are loaded, csid(1) is able to detect type of plugged cards and provide more detailed information.

You should get following output on PC with installed COMBO6X mother card and sfpro add-on card:

$ csid
combo6x sfpro xc2vp20
NOTE:Chips can be different on some cards, so don't worry about differing last part of the csid output

4.2   Setting up NIC Card

nic is main start up script for the NIC Card. Script boots firmware files (according to FIRMWARE_PATH variable from the /etc/liberouter/nic.conf) and setting up card behavior. By default, the card settings are set according to variables from /etc/liberouter/nic.conf configuration file. Optionally you can define your own settings by options given to the script. More detailed information you can get by -h option or from man page:

$ nic -h
$ man nic
NOTE:Both scripts (nic as well as niclkm) are using lsof(8) tool to detect any process using COMBO6(X) device(s). To run scripts properly you have to install lsof(8) tool. lsof(8) is utility to get information about files opened by processes.

4.2.1   Loading kernel modules and running the NIC Card

# niclkm -l
$ nic

To make sure that the firmware was loaded correctly you can run csid(1):

$ csid -s
Board    : combo6x
Addon    : sfpro
Chip     : xc2vp20
LAN ports: 4
Firmware : ok
SW       : 0x41c10200
HW       : 0x00030003
Text     : NIC_NetCOPE
PCI brver: c610.05.09 (2007/11/09 14:30) NetCOPE Bridge
PCI PPC  : [nic Nov  9 2007 21:02:59] 0xdeadbaff -booted- -running-

HW number detected by csid -s reflects firmware version, e.g. HW : 0x00030003 reflects version NIC_03_03.

4.2.2   Stopping the NIC Card and removing kernel modules

To stop running NIC Card and detach drivers, use nic script with the -s option:

$ nic -s

Then you can remove loaded NIC kernel modules by the niclkm script:

# niclkm -r

5   Contacting the NIC team

5.1   Mailing lists

For any questions or general technical support issues, please send mail to the combo6@liberouter.org general mailing list.