Version 1.2.1, 14 July 2010
Copyright © 2009-2010 CESNET, z.s.p.o.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. The license can be found at GNU web page.
Table of Contents
List of Figures
List of Tables
- 2.1. COMBO Motherboard Compatibility
- 2.2. Tested transceivers
- 2.3. Supported Hardware
- 6.1. Data format
- 6.2. Data Format Example
NetCOPE is platform for easy implementation of hardware-accelerated networking applications. It consists of two basic parts: Firmware and software.
Firmware provides an abstraction layer that contains common modules required in networking applications firmware (e.g. Network communication, PCI/PCEe communication, etc.). User can implement and add own modules, translate the design and load the firmware into the card. The basic example is the NIC Card firmware. It is installed in
/usr/share/netcope/fw-{CARD_TYPE}/
There is documentation in the doc directory for each card type. This documentation describes the whole firmware structure and provides step-by-step guide how to build the firmware.
If user doesn't want to build own firmware, the NIC firmware is pre-built in the
/usr/share/mcs/nic/05_06/(version)
directory. To be able to communicate with the hardware card, Linux kernel modules combov2-boot and szedata2_cv2 must be loaded. Selected firmware is loaded into the card by the command
csboot -f 0 firmware.mcs
To get information about the card and the loaded firmware, use command
csid -s
See the NetCOPE First Steps chapter to see more details about card and drivers initialization.
Software part of NetCOPE contains Linux kernel modules, libraries, and several tools for firmware management. Following chapters contain the description of software architecture.
Table of Contents
NetCOPE Platform software works on GNU/Linux OS with 2.6 kernel. It is is distributed in form of several RPM packages. Currently, only CentOS 5 distribution is fully supported.
All scripts are running in /bin/sh shell interpreter so all scripts were created with aim to portability. But on all testing machines the /bin/sh program was actually Bash (/bin/bash). Therefore we recommend using Bash as a default shell.
NetCOPE has been designed to work on any PC-AT compatible computer running GNU/Linux operating system, but it has been tested only on computers with 64-bit (x86-64) processors.
Hardware requirements necessary to run NetCOPE Platform are:
Computer with 64-bit (x86-64) CPU
PCI Express bus
COMBO-LXT155 - PCIe mother card with LX155T-2 chip
COMBOI-10G2 - two 10Gbps interface card
or
COMBOI-1G4 - four 1Gbps interface card
COMBO-LX155T card and one of COMBOI interface cards are connected to each other making unit called "sandwich". It fits into one PCI Express x8 slot, but requires another one because of its size.
The following table shows COMBO card compatibility with some selected motherboards.
Table 2.1. COMBO Motherboard Compatibility
| Motherboard | Chipset | Status | Note |
|---|---|---|---|
| Supermicro X7DBN | Intel 5000P (Blackford) | compatible | Default motherboard used in The Liberouter Project |
| Supermicro H8SMI-2 | nVidia MCP55 Pro | compatible | |
| Gigabyte MA78GM-US2H | AMD 780G | compatible | |
| Supermicro C2SEE-O | Intel G43 + ICH10 | compatible | |
| Supermicro X8STE | Intel 3210 | compatible | |
| Intel S5520UR | Intel X58 | compatible | |
| Intel D945GCCR | Intel ICH7 | incompatible | Chipset fan prevents the card installation into the slot |
The following table lists compatible transceivers that have been tested. We have not found any incompatible transceivers yet. All transceivers satisfying the SFP/XFP MSA specification should work.
Table 2.2. Tested transceivers
| Optical 10 Gbps (10GBASE-SR and 10GBASE-LR) for COMBO-2XFP2 cards |
|---|
| JDSU JXP-01SWAA1 (850 nm) |
| JDSU JXP-01LWAA1 (1310 nm) |
| Finisar FTRX-1411D3 (1310 nm) |
| Agilent (Avago) HFCT-721XPD (1310 nm) |
NetCOPE Platform can be used with COMBO cards described in the following table.
Table 2.3. Supported Hardware
| COMBO-LX155T | An Express PCIx8 mother card belonging to the second generation of COMBO cards. Introduction of the COMBOv2 family was done at XILINX Forum. COMBO-LXT card is equipped with XILINX Virtex5 XC5VLX110T, QDRII RAM and socket for DDR2 SODIMM memory. Addon cards can by connected either through Low Speed Connectors or high speed Interface Connectors. | |
| COMBOI-10G2 | Two port 10Gbps interface card for COMBOv2 family cards. | |
| COMBOI-1G4 | Four port 1Gbps interface card for COMBOv2 family cards. |
Note
You can find more information about COMBO cards at http://www.liberouter.org/hardware.php.
Connect COMBO-LX155T mother card with one COMBOI interface card together (see Figure 2.1, “COMBO-LX155T and COMBOI-10G2 connected together”). This connection makes unit called "sandwich". It fits into one PCI Express x8 slot, but requires another one because of its size.
Carefully plug in the "sandwich" into your PCI slot, while the computer is turned off.
You should test connection between the card and your PC after successfull installation. You can use lspci(8) utility for this purpose.
Note
lspci(8) is an utility showing information about all PCI buses in the system and all devices connected to them. For correct recognition of the COMBO-LXT card you need to update PCI ID Database used by lspci(8) or download pciutils-2.2.2 (program collection containing lspci(8) or later. If the lspci(8) output contains the following line your COMBO-LXT card is connected properly:$lspci-d18ec:09:00.0 Ethernet controller: Cesnet, z.s.p.o. COMBO-LXT155 (rev d8)
Numbers at the beginning of any output line could be different.
The Liberouter project has a common centralized RPM repository for all projects. It is available on URL: https://www.liberouter.org/repo/STABLE/x86_64/. Repository works over secured HTTPS channel. To obtain packages from it, you have to authorize yourself with login and password which you have received as our client. (Eventually, use login and password for SVN account.)
To make the standard tool yum(1) work properly with
our repository, you have to setup some configuration.
Add a new file liberouter.repo
to directory /etc/yum.repos.d/
with content:
[liberouter]
name=Liberouter RPM repository
baseurl=https://LOGIN:PASS@www.liberouter.org/repo/STABLE/x86_64/
enabled=1
gpgcheck=1
gpgkey=https://www.liberouter.org/repo/RPM-GPG-KEY-LiberouterNote
Instead of LOGIN and PASS write your real client login and password. To hide these private information you should change file mode bits by chmod(1):
# chmod 600 /etc/yum.repos.d/liberouter.repo
Note
Liberouter packages are digitally signed using GPG. Public key with GPG ID: 170EC4C1 (fingerprint FB55 3BB6 1571 0361 04CF 766A CAAF 4A06 170E C4C1) used to verify installing Liberouter packages is stored in file https://www.liberouter.org/repo/RPM-GPG-KEY-Liberouter.
Now you can check your configuration using yum(1):
# yum repolist
...
liberouter Liberouter RPM repository enabled
...
NetCOPE software is distributed in a form of several RPM packages. Currently RPM packages only for CentOS 5 distribution are supported.
To install NetCOPE packages use YUMs groupinstall command:
# yum groupinstall netcope-platform-combov2-10g2or
# yum groupinstall netcope-platform-combov2-1g4according to installed card.
Here follows the list of used installation paths with their description.
/lib/modules/$(uname -r)/*/*.ko ...................... LKM drivers /usr/bin/ ............................................ binary tools /usr/include/ ........................................ header files /usr/lib/pkgconfig/ .................................. pkg-config libraries /usr/lib/ ............................................ libraries /usr/share/man/manX/ ................................. manual pages, eg. man netcope-platform(7), or man netcope(1) /usr/share/mcs/ ...................................... firmware design files /usr/share/netcope/fw-combov2-10g2/ .................. COMBOv2-10G2 card firmware files /usr/share/netcope/fw-combov2-10g2/doc/ .............. COMBOv2-10G2 card documentation /usr/share/netcope/fw-combov2-1g4/ ................... COMBOv2-1G4 card firmware files /usr/share/netcope/fw-combov2-1g4/doc/ ............... COMBOv2-1G4 card documentation /usr/share/netcope/netcope-platform-handbook/ ........ NetCOPE platform handbook documentation /usr/share/netcope/sw-examples/ ...................... NetCOPE platform software examples /usr/share/netcope/sw-examples/doxygen/ .............. Doxygen documentation for NetCOPE platform libraries (libcombo, libcommlbr, libsze2, libpcap-sze) /usr/share/netcope/sw-examples/README ................ NetCOPE platform software examples README (compilation how-to) /usr/share/netcope/README.html ....................... NetCOPE basic README /usr/share/netcope/RELNOTES.html ..................... NetCOPE platform release notes
If you want to allow non-privileged user to use combo cards, add him to combo-rw group:
# usermod -G $(id -G <user> | sed 's/ /,/g'),combo-rw <user>
Or simply edit /etc/group file, where you have to find line begining with
the 'combo-rw' string and to the end of this line add required user login.
So the result should look like this for the user 'bfu':
combo-rw:x:10001:bfu
Note
group ID can differ on your system
Table of Contents
Software layer of NetCOPE platform takes care of cooperation between hardware acceleration card and operating system and also provides software interfaces to work with hardware acceleration card and to use its functionality.
Software architecture is composed of three basic layers. Communication between acceleration card and operating system is done by system drivers. NetCOPE libraries provide interface for creating applications using acceleration card. Tools layer consists of a few utilities capable of configuration of basic hardware platform components and utilities for testing and debugging. This architecture is visualized on the following picture.
Platform software can be divided into two logical groups. First and basic one covers basic input/output operations with hardware card, card initialization, boot process for loading firmware, monitoring and configuration of hardware components. Second logical group named "SZE" provides software support for fast DMA transfers of user specific data to user applications.
System drivers create basic and low-level connection among operating system, hardware card and firmware.
- combo6
system drivers to control Combo6, Combo6X, ML555 and ComboV2 cards. Provide basic input/output for follow-up libraries.
- szedata2*
system drivers to take care of fast DMA transfers.
The basic system driver for ComboV2 cards is called combov2_boot, which is used to communicate with the firmware after system power-up. Next important driver is szedata2, which performs fast DMA transfers between firmware and host RAM. To enable it, simply load the kernel module:
$ modprobe szedata2
To load the module without support for standard network interface (see later), use the no_eth parameter. This may help to increase the system throughput.
$ modprobe szedata2 no_eth=y
szedata2_cv2 accomodates the driver to work with ComboV2 family cards. Among other, it decides about the size of software buffers for DMA data. User may set it to arbitrary number of blocks, where one block size is always 4 KiB. For example:
$ modprobe szedata2_cv2 block_count_rx=32768 block_count_tx=16384
In this example, the size of receive buffers is set to 128 MiB each, and the size of transmit buffers is set to 64 MiB each.
Libraries are next layer above system drivers. They provide system drivers functionality in more comfortable, user level way. They create abstraction of software control of hardware components.
- libcommlbr
The lowest library. It contains macros, structures and functions for debugging, version control and also frequently used routines in most of NetCOPE tools such as cmd parameters processing, data types conversions, time functions, dumping buffers, working with PCAP files and structures.
- libcombo
Library provides interface for firmware booting, its initialization, input/output operations, work with firmware and its components and mapping firmware components address space into user applications.
- libsze2
Library provides low-level and higher user level interface to send and receive data through szedata2 interface.
- libpcap-sze
Libpcap library enriched with capability of working over szedata2 interface. It allows standard applications such as tcpdump, ethereal or snort to work without any changes with benefits of fast DMA transfers performance.
Tools create command line interface for booting, monitoring, configuration, debugging and testing of firmware components.
- Basic Public Tools
csxtool, csboot, csbus, csid - tools for booting firmware, input/output and for getting information about currently loaded firmware.
- Public Tools
ibufctl, obufctl, phyterctl, tsuctl - tools for monitoring and configuration platform components, including tasks such as network interface speed, duplex, autonegotiation configuration and precise timestamp module configuration.
- Szedata2 Development Tools
sze2loopback, sze2write - tools for testing of correctness and throughput of szedata2 and libpcap-sze2 interfaces.
-
NetCOPE Platform User Handbook
This document.
-
NetCOPE Firmware Guide
Description of firmware structure, guides to modify and build user designs from examples. Can be found in
/usr/share/netcope/fw-{CARD_TYPE}/doc -
Pcap man page
Pcap library documentation is described in pcap man page accessible by command "man pcap".
-
Man Pages
NetCOPE platform is briefly described in netcope man page accessible by command "man netcope". There is also list of common NetCOPE tools, from which the most have their own manual pages or documentation.
-
Software examples
Examples showing basic work with libcombo, libsze2, libpcap libraries. Can be found in
/usr/share/netcope/sw-examples/
directory.
-
Doxygen documentation
Doxygen documentation is provided for all NetCOPE libraries - libcommlbr, libcombo, libsze2, libpcap-sze. Can be found in
/usr/share/netcope/sw-examples/doxygen/
directory.
To configure the hardware and firmware, set of configuration tools is available:
-
csid
This tool reads info about the card and its current firmware.
-
csboot
Tool for changing the card firmware (booting).
-
ibufctl, obufctl
Two tools to probe and change the state of network input and output buffers.
-
phyterctl
This tool probes and changes the state of phyters and transcievers.
For all these tools, please use the -h switch to get more detailed info about their use.
Standard network interface - NIC over NetCOPE acts in system as ordinary network interface called cethX. Received packets are passed from driver to operating system kernel and further through TCP stack to target application. Alternatively PCAP library, part of operating system, can be used to access packets in raw format avoiding passage through TCP stack (typical for monitoring applications as Tcpdump, Wireshark, Snort etc.). Advantage of this interface is its universality and possibility of using existing applications. Standard tools, such as ifconfig, may be used. Disadvantage is low throughput caused by passage through operating system kernel which requires disassembling packets into segments without exploiting advantages of aggregation.
NetCOPE provides the szedata2 general interface for block transfers to applications. Communication takes place directly between driver and userspace application without passage through Linux TCP stack. Format of passed data is not restricted by operating system kernel and therefore this interface can benefit from using aggregated packets with control information, timestamps etc. appended. Various data (depending on accelerator purpose) such as netflow records, information extracted from packet headers, etc. can be transferred. NIC over NetCOPE uses this interface to transfer aggregated packets. In addition, PCAP library is adjusted to work within this interface, communicating directly to driver, avoiding passage through operating system kernel. Main advantage of this approach is high throughput using aggregation of smaller packets reducing DMA transfers overhead.
Table of Contents
The Szedata2 interface transfers any data (not only packets) in segments of variable length from the acceleration card to the host RAM or backwards. Before transfer, data segments are aggregated to bigger blocks to reduce PCI/PCIe bus overhead.
In this section, format of data passed to software through Szedata2 interface is described.
One block typically contains more data segments of variable length. Segment contains hardware service data (hardware header) and content of one packet (or any other user data). Hardware header size depends on the application. Every application can specify its hardware header structure according to its needs, only the first four bytes must have fixed format. Length of whole segment is aligned to 8 bytes, although real length of data can be smaller - this empty end gap must be skipped by application.
Table 6.1. Data format
| Offset (bytes) | Size (bytes) | Description |
|---|---|---|
| 0x00 | 0x02 | Segment size in bytes (Big Endian). Size of useful data (whole szedata2 packet including this whole header, without end align zeros gap) |
| 0x02 | 0x02 | Size of the rest of the hardware data, not including the first four bytes (Big Endian) |
| 0x04 | Size of hardware data | Hardware header (application specific) |
| 0x04 + Size of hardware data | ALIGN(0x04 + Size of hardware data, 8) - (0x04 + Size of hardware data) | Header align gap (if any) |
| ALIGN(0x04 + Size of hardware data, 8) | Segment size in bytes minus offset | Segment data (application specific, eg. packet content), beginning aligned to 8 bytes |
| Segment size | ALIGN(Segment size, 8) - Segment size | End align zeros gap (if any) |
| ALIGN(Segment size, 8) | ... | Next szedata2 packet |
Whole length of szedata2 packet is ALIGN(Segment size, 8).
Table 6.2. Data Format Example
| Offset (bytes) | Size (bytes) | Value | Description |
|---|---|---|---|
| 0x00 (0) | 0x02 (2) | 0x44 (68) | Segment size is 68 bytes |
| 0x02 (2) | 0x02 (2) | 0x03 (3) | Hardware header data size is 12 bytes |
| 0x04 (4) | 0x02 (2) | 0x00 0x02 | Input interface mask (Bit at index 1 is set, so the packet comes from interface 1) |
| 0x06 (6) | 0x02 (2) | 0x00 0x00 | Unused hardware header space |
| 0x08 (8) | 0x08 (8) | 0x4C 0x1D 0x73 0x3F 0x67 0x45 0x23 0x10 | Timestamp in Unix format. This value corresponds do the date and time 2010-06-20 01:47:43,403398692. |
| 0x10 (16) | 0x34 (52) | 0x00 0x01 ... 0x33 | Packet data |
| 0x44 (68) | 0x04 (4) | 0x00 0x00 0x00 0x00 | Segment alignment |
7 6 5 4 3 2 1 0 00 00 00 02 00 0C 00 44 4C 1D 73 3F 67 45 23 10 07 06 05 04 03 02 01 00 0F 0E 0D 0C 0B 0A 09 08 17 16 15 14 13 12 11 10 1F 1E 1D 1C 1B 1A 19 18 27 26 25 24 23 22 21 20 2F 2E 2D 2C 2B 2A 29 28 00 00 00 00 33 32 31 30
This example also shows the format of hardware header in NIC over NetCOPE firmware.
Libsze2 library provides low level and high level functions to work with Szedata2 interface. See doxygen documentation for further reading and API documentation.
receives, or resends received packets through szedata2 interface
The sze2loopback tool is capable of reading received packets from szedata2 interface and also of resending through this interface. Default action is to act as a loopback - send everything received from selected RX interfaces (-r) to specified TX interface (-t). Tool prints progress as 'R' and 'W' characters after received/sent specified count of packets (-I switch). Tool works either with lower level libsze2 library (default) or uses higher libpcap (-P switch) library (compiled with szedata2 support). Besides loopback mode, tool can work either only in RX (receiver -R) or TX (sender -W) mode. Contents of packets can be printed (-S switch). For debugging purposes contents of szedata2 packets (data structures received from driver) can be printed. Whole, hardware or software part of szedata2 packet can be specified (-AHS switches).
Please, see the output of sze2loopback -h for details.
sends packets from pcap dump file through szedata2 interface
The sze2write tool reads packets from pcap dump file (e.g. acquired by tcpdump -w) and sends them through szedata2 interface using libsze2 library (not pcap functions). It prints number of read packets and captured portion of packet from dump file.
Please, see the output of sze2write -h for details.
Well known libpcap library has been modified to support Szedata2 interface as regular NIC. It allows to use applications such as tcpdump, wireshark, nprobe or fprobe over NetCOPE and create new applications using PCAP API.
Detailed howtos and examples for working with libpcap API can be found at tcpdump webpages (www.tcpdump.org).
To check that you have libpcap with szedata2 support run:
# rpm -qi libpcap-sze
and in Description part of the output you should see something like this:
Description : Packet-capture library LIBPCAP 1.1.1 with szedata2 support
If the package is not present, you can install it manually using the following command:
# yum install libpcap-sze
Note
Don't forget to set up yum(1)'s repository configuration to include official Liberouter repository as described in Obtaining Packages section.
When libpcap szedata2 support is installed properly, compile it as with any other library.
You should have this line in your source file:
#include <pcap.h>
For dynamic version (libpcap.so) add -lpcap to compiler flags:
gcc read.c -lpcap -o read ldd read libpcap.so => /usr/local/lib/libpcap.so (0xb7f2e000) libsze2.so.0 => /usr/local/lib/libsze2.so.0 (0xb7f16000) libcommlbr.so.0 => /usr/local/lib/libcommlbr.so.0 (0xb7f10000) libdbus-1.so.3 => /usr/lib/libdbus-1.so.3 (0xb7edf000) ...
For static version (libpcap.a) add libpcap.a as another file to link.
gcc -c read.c -o read.o gcc read.o /usr/lib/libpcap.a -o read
Check if application we want to use is compiled with dynamic version of libpcap library supporting SZE2 (linking libsze2 library):
$ldd $(which tcpdump)libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x0000003dbaa00000) libc.so.6 => /lib64/libc.so.6 (0x0000003ec1c00000) libsze2.so.1 => /usr/lib64/libsze2.so.1 (0x0000003dba600000) /lib64/ld-linux-x86-64.so.2 (0x0000003ec1800000) ...
If currently installed application doesn't use dynamic version of libpcap library, you can try to recompile it or (as in case of the tcpdump) you can download its package from the Liberouter repository.
#yum update tcpdump ... ================================================================= Package Arch Version Repository Size ================================================================= Updating: tcpdump x86_64 14:4.0.0-902 liberouter 327 kRun application with szedata interface as parameter szeX (X is number of card).
tcpdump -i sze0
Optionally particular port can be specified with szeX:N or szeX.N parameter (N is number of port - NIC interface).
tcpdump -i sze0:2
Table of Contents
1G or 10G interfaces are supported. This depends on the type of the card. 10M and 100M interface speeds on the 1G card are supported, but manual switching is required. At the card with 10G interfaces, no speed switching is possible.
phyterctl(1) is a tool used to display and change configuration of interfaces available on COMBO add-on cards. The tool displays information about link status, resolved speed or duplex mode on link. In a normal operation mode, use phyterctl only to probe interfaces status. For speed selection, use the ibufctl(1) tool.
Example of phyterctl(1) listing:
$ phyterctl
Settings for card 0 (device /dev/combosix/0):
------------------------------ Interface 0 ---
Phyter vendor VITESSE
Phyter model VSC8486 10GbE PHY (rev 3)
Speed 10 Gb/s
Receive signal Loss
TX status Enabled
TX/RX Fault 0/1
Link status Down
------------------------------ Interface 1 ---
Phyter vendor VITESSE
Phyter model VSC8486 10GbE PHY (rev 3)
Speed 10 Gb/s
Receive signal Detected
TX status Enabled
TX/RX Fault 0/0
Link status Up
ibufctl(1) is used to display and change configuration of IBUF components in COMBO designs.
Example of ibufctl(1) listing:
$ ibufctl -i0
-------------------------------------- IBUF Status ----
Interface number : 0
IBUF : ENABLED
PACODAG overflow occurred : False
DFIFO overflow occurred : True
IBUF speed : 10Mb/s
------------------------ IBUF Packets/Frames Stats ----
Packets : 653730
Received : 653730
Discarded : 0
Buf overflow : 0
Error packets : 0
------------------------------------ IBUF Settings ----
Frame error from GMII [1] : enabled
CRC check [2] : enabled
Minimum frame length [4] : enabled
* length : 64
MTU frame length [8] : enabled
* length : 1526
MAC address check [16] : enabled
* mode : [0x0] promiscuous
To change interface speed, use -s switch of ibufctl. It is recommended to set the desired interface speed every time after FPGA design reboot, because there are hardware components which are not reset during FPGA reboot. At the card with 10G interfaces, no speed switching is possible.
To change MAC address of input interface, use the ibufctl tool. Each interface may have up to 16 different input MAC addresses. MAC address filtering may be turned on or off.
To change MAC address of output interface, use the obufctl tool. Each interface is able to insert its MAC address into outgoing packets.
More information can be found in the phyterctl(1), ibufctl(1) and obufctl(1) man pages.
This chapter shows how to use the Network Interface Card on the NetCOPE platform. We start with the card detection:
$lspci-d18ec:09:00.0 Ethernet controller: Cesnet, z.s.p.o. COMBO-LXT155 (rev d8)
18ec is the vendor number assigned to CESNET. 09:00.0 is the number of chipset PCI port which the card is connected to in this particular example. We can see that the card is installed in the PCIe slot. Next we make sure that necessary Linux kernel modules are loaded:
$ lsmod | egrep "combov2_boot|szedata2_cv2" szedata2_cv2 12948 0 szedata2 16268 1 szedata2_cv2 combov2_boot 4600 0 combov2 16256 2 szedata2_cv2,combov2_boot combo6core 22836 4 szedata2_cv2,combov2,combo6x,combo6
If combov2_boot or szedata2_cv2 modules are not present, load them (root account must be used to load/unload kernel modules):
# modprobe szedata2_cv2 # modprobe combov2_boot
Now we are ready to access the card firmware. We begin with reading the basic card identification (ComboLXT155 with 10G2 addon card shown, identified as 2x10G1):
$ csid -s Board : combo Subtype : LXT155 S/N : 8100352 Addon0 : 10G1 Chip0 : n/a S/N0 : 8100382 Addon1 : 10G1 Chip1 : n/a S/N1 : 8100382 Channels : 82/33 (RX/TX) Firmware : ok SW : 0x1111dead HW : 0x00000000 Text : PCI brver: 6d05.01.02 (2009/03/24 14:03)
This is the identification of the default design after power-up. Now we load our firmware and check the firmware identification:
$ csboot -f 0 /usr/share/mcs/nic/05_09/cv2_10G2/nic_cv2_155.mcs
$ csid -s
Board : combo
Subtype : LXT155
S/N : 8100325
Addon0 : 10G1
Chip0 : n/a
S/N0 : 8100382
Addon1 : 10G1
Chip1 : n/a
S/N1 : 8100382
Channels : 2/2 (RX/TX)
Firmware : ok
SW : 0x41c10509
HW : 0x00060000
Text : NIC_CV2_10G2
PCI brver: 6d05.01.04 (2009/06/10 14:24)
Device [combo6] szedata2
(0x41c10400-0x41c105ff) {}: active
Device [combo6] szedata2
(0x41f10100-0x41f101ff) {}: inactive
Device [combo6] szedata2
(0xf1010100-0xf10101ff) {}: inactive
We can see that the design identification has changed. Now, to enable receiving packets into the card, enable input buffer 0 and check its status:
$ ibufctl -i 0 -e 1 $ ibufctl -i 0 -------------------------------------- IBUF Status ---- Interface number : 0 IBUF : ENABLED PACODAG overflow occurred : False DFIFO overflow occurred : False IBUF speed : 10Gb/s ------------------------ IBUF Packets/Frames Stats ---- Packets : 0 Received : 0 Discarded : 0 Buf overflow : 0 Error packets : 0 ------------------------------------ IBUF Settings ---- Frame error from GMII [1] : enabled CRC check [2] : enabled Minimum frame length [4] : enabled * length : 64 MTU frame length [8] : enabled * length : 1526 MAC address check [16] : enabled * mode : [0x0] promiscuous
Input buffer is now enabled and packets may flow into the card. Similarly, to enable sending packets from the card, enable output buffer 0 and check its status:
$ obufctl -i 0 -e 1 $ obufctl -i 0 ------------------ OBUF Status ---- OBUF : ENABLED OBUF speed : 10Gb/s ---- OBUF Packets/Frames Stats ---- Packets : 0 Transmitted : 0 Err packets : 0 ---------------- OBUF Settings ---- MAC address : 00:00:00:00:00:00
Output buffer is now enabled and packets may be transmitted from the card. Next, we will enable receiving packets into the software and sending them back into the card. We will use the sze2loopback tool to read packets continuously from all ports of the card, and send them back on the interface 0:
$ sze2loopback -I 10000 -t 0
Now start sending some traffic into the card with a third-party device. We have enabled input buffer 0, so the traffic must go into port 0 on the card, which is the closest one to the PCIe slot.
After a few seconds, stop the traffic into the card and press Ctrl+C to interrupt the sze2loopback tool. Every character 'R' in the sze2loopback output represents 10000 packets received into the software, and every W represents 10000 packets sent into hardware because we have used -I 10000 switch.
RWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWsze2loopback: poll failed: Interrupted system call sze2loopback: Poll interrupted Recieved: 272889 Written : 272889 $ ibufctl -i 0 -------------------------------------- IBUF Status ---- Interface number : 0 IBUF : ENABLED PACODAG overflow occurred : False DFIFO overflow occurred : False IBUF speed : 10Gb/s ------------------------ IBUF Packets/Frames Stats ---- Packets : 272889 Received : 272889 Discarded : 0 Buf overflow : 0 Error packets : 0 ------------------------------------ IBUF Settings ---- Frame error from GMII [1] : enabled CRC check [2] : enabled Minimum frame length [4] : enabled * length : 64 MTU frame length [8] : enabled * length : 1526 MAC address check [16] : enabled * mode : [0x0] promiscuous $ obufctl -i 0 ------------------ OBUF Status ---- OBUF : ENABLED OBUF speed : 10Gb/s ---- OBUF Packets/Frames Stats ---- Packets : 272889 Transmitted : 272889 Err packets : 0 ---------------- OBUF Settings ---- MAC address : 00:00:00:00:00:00
As we can see, input buffer received 272889 packets and all of them were transferred into the software. At the same time, the sze2loopback tool was continuously sending all packets back into the card and the output buffer sent all packets into the network interface 0.
Other option of receiving packets is use of Standard network interface. These interfaces are called ethX and are managed by standard ifconfig(8) tool. Any standard application using UNIX sockets should be compatible with cethX interfaces. Input and output buffers, however, are still controlled by ibufctl and obufctl tools. Example of listing all network interfaces:
$ ifconfig -a
ceth0 Link encap:Ethernet HWaddr 00:11:17:00:00:00
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::211:17ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:45 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:11660 (11.3 KiB)
ceth1 Link encap:Ethernet HWaddr 00:11:17:00:00:01
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
eth0 Link encap:Ethernet HWaddr 00:30:48:CB:EE:3A
inet addr:147.251.21.130 Bcast:147.251.21.255 Mask:255.255.255.0
inet6 addr: 2001:718:801:330::93fb:1582/60 Scope:Global
inet6 addr: fe80::230:48ff:fecb:ee3a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:68787 errors:0 dropped:154 overruns:0 frame:0
TX packets:28801 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:100
RX bytes:47505528 (45.3 MiB) TX bytes:17841627 (17.0 MiB)
Memory:dc620000-dc640000
eth1 Link encap:Ethernet HWaddr 00:30:48:CB:EE:3B
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Memory:dc660000-dc680000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:99 errors:0 dropped:0 overruns:0 frame:0
TX packets:99 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10588 (10.3 KiB) TX bytes:10588 (10.3 KiB)
sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
In this example, ceth0 interface is up, and ceth1 is down. To enable interface ceth1 (superuser privilegies needed):
# ifconfig ceth1 192.168.1.2 netmask 255.255.255.0 up