#     #  #######  #     #  #######     #
##    #  #        ##   ##  #          # #
# #   #  #        # # # #  #         #   #
#  #  #  #####    #  #  #  #####    #     #
#   # #  #        #     #  #        #######
#    ##  #        #     #  #        #     #
#     #  #######  #     #  #######  #     #

This file describes the Nemea system in detail. To see more general information,
please have a look at http://www.liberouter.org/technologies/nemea or the CESNET
technical report 6/2013: V. Bartos, M. Zadnik, T. Cejka: "Nemea: Framework for
stream-wise analysis of network traffic".

The text is divided as follows:
Section 1 contains the installation guide and the list of dependencies.
Section 2 contains introduction to usage of the Nemea system, it can be used
as a quick start guide.
Section 3 describes control and monitoring module called Supervisor. Supervisor
keeps the whole system running and monitors the workload of running modules.
Section 4 provides a brief description of the modules that are included
in the Nemea distribution package.

------------------------------------------------------------------------------
1. Installation (responsible for content: cejkat@cesnet.cz)
------------------------------------------------------------------------------
The Nemea system consists of the Nemea framework, which is composed of libtrap,
libunirec and libnemea-common libraries, and Nemea modules. The whole system
is based on GNU/Autotools build system that makes dependency checking and
building process much more easier.

Libraries from the Nemea framework can be compiled without any special
dependencies. However, some of the included modules depend on libraries:
  libidn (should be available in distribution repositories)
  libnfdump (http://sourceforge.net/projects/libnfdump/)
In order to build these modules, there must be development packages or source
codes available on the system. Installation of dependencies is out of scope of
this text.

The first step in compilation from the Nemea distribution package is the
execution of `configure` script. The script supplies various possibilities of
configuration and uses some environmental variables that influence the build
and compilation process. The list of supported settings with the description
of environmental variables are available by the help of configure script, see:
  ./configure --help

The configure script allows users to set target paths for installation of
the results of the build process. The important parameters for changing paths are:
  --prefix, --libdir, --bindir, --docdir, --sysconfigdir

The configure script enables silent mode of make(1) by default. To disable this
feature, please pass the parameter:
  --disable-silent-rules

After successful run of the configure script, package should be ready for
compilation. The compilation can be started by:
  make
The make(1) tool has various parameters, to build the Nemea package faster on
multicore systems, we recommend to use parameter -j with the number of jobs
that should be run in parallel.

When compilation ends without any error, the package can be installed into paths
that were set during configuration. It is recommended NOT to change target paths
by passing variables to make(1).
The installation can be done by:
  make install

Congratulations, the whole Nemea system should be installed right now... :-)


------------------------------------------------------------------------------
2. Quick Start Guide (responsible for content: xkrobo01@stud.fit.vutbr.cz)
------------------------------------------------------------------------------
This section shows how to manually start Nemea modules. Module description and
specific parameters could be shown by running module with parameter "-h":
  nfdump_reader -h

There are two possible data inputs for Nemea modules:
  a) from nfdump file (static data)
  b) from IPFIXcol (https://www.liberouter.org/ipfixcol/) with UniRec plugin
     (real-time data).
	
a) Data from nfdump files could be read and sent to Nemea by "nfdump_reader"
module. Following command will send records from file "nfcapd.201406001" to TCP
interface on port 9988 in UniRec format:
  nfdump_reader -i "t;9988" /data/nfcapd.201406001

Another module then could capture and process this data, e.g. DNS amplification
detector:
  dns_amplification -i "tu;localhost,9988;DNS_amp"
  
After executing both commands, DNS amplification detection on data from given
file will be done. Possible attacks will be reported on Unix socket "DNS_amp".

b) Real-time data from IPFIXcol could be provided to Nemea also. First run
IPFIXcol, e.g.:
  ipfixcol -d -c /data/configs/startup.xml

File "startup.xml" should contains configuration for IPFIXcol plugin, among
others there will be settings for output UniRec format and output interface.
Example of such configuration file for UniRec plugin is at the end of this
section. Command for running Nemea module with input from IPFIXcol (e.g. on TCP
port 9966) remains same as with input from "nfdump_reader":
  dns_amplification -i "tu;localhost,9966;RT_DNS_amp"
  
Another examples of starting Nemea modules follows:
ex1) This example starts two "nfdump_readers" on two different files. Records
from both files are then merged into one stream and sent on TCP interface,
port 9920:
  nfdump_reader -i "t;9911" /data/link1/nfcapd.201406001
  nfdump_reader -i "t;9912" /data/link2/nfcapd.201406001
  merger -i "ttt;localhost,9911;localhost,9912;9920" -n 2

ex2) This more complex example start one "nfdump_reader" on multiple files. All
files are read sequentially and sent on output interface. Data from
"nfdump_reader" are then anonymized and sent to "hoststatsnemea" detector and
to "flowcounter". Reports from detector are then stored to CSV file via
"logger":
  nfdump_reader -i "u;HS_src" /data/0601/nfcapd.0000 /data/0601/nfcapd.0005
    /data/0601/nfcapd.0010 /data/0601/nfcapd.0015
  anonymizer -i "uu;localhost,HS_src;HS_an" -k 0AnonymizationKeyWithLengthof32B
  flowcounter -i "u;localhost,HS_an" -p 100000
  hoststatsnemea -i "uu;localhost,HS_an;HS_report" -F
  logger -i "u;localhost,HS_report" "EVENT_TYPE,TIME_FIRST,TIME_LAST,SRC_IP,
    DST_IP,SRC_PORT,DST_PORT,PROTOCOL,EVENT_SCALE"
    
note: In hoststatsnemea configuration file should be "port-flowdir = 1".

Example of configuration file for UniRec plugin for IPFIXcol:
<?xml version="1.0" encoding="UTF-8"?>
<ipfix xmlns="urn:ietf:params:xml:ns:yang:ietf-ipfix-psamp">

  <collectingProcess>
    <name>UDP collector</name>
    <udpCollector>
      <name>Listening port 4740</name>
      <localPort>4740</localPort>
      <localIPAddress></localIPAddress>
    </udpCollector>
    <exportingProcess>UniRec output</exportingProcess>
  </collectingProcess>

  <exportingProcess>
    <name>UniRec output</name>
    <destination>
      <name>Make unirec from the flow data</name>
      <fileWriter>
        <fileFormat>unirec</fileFormat>
        <!-- Default interface -->
        <interface>
          <type>t</type>
          <params>9966,16</params>
          <ifcTimeout>0</ifcTimeout>
          <flushTimeout>10000000</flushTimeout>
          <bufferSwitch>1</bufferSwitch>
          <format>DST_IP,SRC_IP,BYTES,LINK_BIT_FIELD,TIME_FIRST,TIME_LAST,
		    PACKETS,?DST_PORT,?SRC_PORT,DIR_BIT_FIELD,PROTOCOL,?TCP_FLAGS</format>
        </interface>
      </fileWriter>
    </destination>
  </exportingProcess>
</ipfix>
------------------------------------------------------------------------------
3. Manage Nemea modules efficiently (responsible for content: svepemar@fit.cvut.cz)
------------------------------------------------------------------------------
The Nemea system contains a module called Supervisor. This module allows user
to configure and monitor Nemea modules. User specifies modules in XML configuration
file, which is input for the Supervisor.

Example configuration of one module called flowcounter:

<module>
  <enabled>false</enabled>
  <params></params>
  <name>flowcounter</name>
  <path>../modules/flowcounter/flowcounter</path>
  <trapinterfaces>
    <interface>
      <note></note>
      <type>TCP</type>
      <direction>IN</direction>
      <params>localhost,8004</params>
    </interface>
  </trapinterfaces>
</module>

Every module contains unique name, path to binary file, parameters, enabled flag
and trap interfaces. Enabled flag tells Supervisor to run or not to run module after
startup. Every trap interface is specified by type (TCP, unixsocket or service),
direction (in or out), parameters (output interface: address + port; input
interface: port + number of clients) and optional note.

User can do various operations with modules via Supervisor:
 - start or stop all modules,
 - start or stop one module,
 - display modules status,
 - display loaded configuration,
 - reload configuration (initial XML file or another XML file).

Supervisor monitors the status of all modules and if needed, modules are auto-restarted.
Every module can be run with special "service" interface, which allows Supervisor to get
statistics about sent and received messages from module.
Another monitored event is CPU usage of every module.
These events are periodically monitored.

Supervisor also provides logs with different types of output:
 logs#1 - stdout and stderr of every started module
 logs#2 - supervisor output - this includes modules events (start, stop, restart,
          connecting to module, errors etc.) and in different output file statistics
          about messages and CPU usage.

To get more information about Supervisor, read nemea/supervisor/README.

------------------------------------------------------------------------------
4. Brief Description of modules (responsible for content: zadnik@fit.vutbr.cz)
------------------------------------------------------------------------------

Anonymizer (anonymizer)
-----------------------
  Receives UniRec messages from input libtrap interface, modifies IP addresses 
  included in the UniRec using cryptography and sends them via output libtrap interface.

ASTUTE (astute)
---------------
  The module implements the ASTUTE method published in:

    Silveira, Fernando, et al. "ASTUTE: Detecting a different class of traffic anomalies."
    ACM SIGCOMM Computer Communication Review 40.4 (2010): 267-278.

  The method groups network flows into 5-minute time windows and compares flows from two
  following time windows. The output is the 6-tuple for different aggregation of flows.

Blacklist filters
-----------------
  The Nemea package contains a set of modules that use blacklist filtering of various
  application protocols.

  The modules are:
      * IP Blacklist Filter (ipblacklistfilter)
      * DNS Blacklist Filter (dnsblacklistfilter)
      * URL Blacklist Filter (urlblacklistfilter)

Counters and statistics
-----------------------
  The Nemea package contains a set of modules that computes basic statistics of the incoming
  messages.

  The modules are:
      * IPv6 Statistics (ipv6stats)
      * HTTP Counter(http_counter)
      * FlowCounter (flowcounter)
      * DNS counter (dns_counter)
      * SIP Counter (sip_counter)
      * SMTP Counter (smtp_counter)


DNS Amplification detector (dns_amplification)
----------------------------------------------
  For each pair of communicating IP addresses with a specific communication port,
  the module can detect an attack using these features:
      * sizes of requests and responses are homogeneous
      * sizes of requests and responses are asymmetric
      * requests rate
  Historical data are periodically deleted.

Entropy module (entropy_module)
-------------------------------
  The module receives UniRec messages and computes the set of entropy values from
  combinations of UniRec fields.
  The module can compute 4 or 11 different entropy values, additional entropy values can be enabled
  module's parameter.

Example Nemea Module (example_module)
-------------------------------------
  The example module shows the usage of Nemea framework for a module creation.

FlowDirection (flowdirection)
-----------------------------
  The module tries to recognize the direction of flow: Single Flow, Response, Request.

HeartBleed Detector (heartbleed_detector)
-----------------------------------------
  Detection of exploit of the vulnerability in some versions of the OpenSSL library.
  The module is able to detect HeartBleed attack using extended data from monitoring probe
  that contain additional fields from application protocol.

  The detection works on encrypted communication as well.

HostStats Nemea (hoststatsnemea)
--------------------------------
  Hoststatsnemea is the Nemea module that contains functionality
  of the HostStats project (http://sourceforge.net/projects/hoststats/).

  The module can detect e.g. (D)DOS or scan.

IP Spoofing Detector (ipspoofingdetector)
-----------------------------------------
    IPSpoofingDetector module implements function for checking the received
    IPFIX data for Source IP address spoofing. Implementation of the detection 
    method is based on article:

      Kovacik Michal, Kajan Michal, Zadnik Martin:
      "Detecting IP-spoofing by modelling history of IP address entry points"

Logger (logger)
---------------
  Logger is a useful module for export of incoming UniRec messages into CSV file.

LogReplay (logreplay)
---------------------
  LogReplay is a complement module for Logger. It can read CSV files and send
  the data in them in form of UniRec messages via libtrap output interface.

Merger (merger)
---------------
  The merger module can be used for re-transmission messages from different input
  interfaces via single output interface.
  It is suitable to merge messages from different message streams of Nemea modules
  or monitoring probes.

Nfdump Reader (nfdump_reader) / Nfdump Reader test version (nfdump_reader_test)
-------------------------------------------------------------------------------
  Module can be used for replay of NFDUMP files created by nfcapd(1).
  The module uses the libnfdump shared object that is available at:

    http://sourceforge.net/projects/libnfdump/

  Flow records from file(s) are resent in UniRec format via libtrap output interface.

  Nfdump_reader_test differs from nfdump_reader in the way of load and resend of messages.
  (nfdump_reader_test loads the whole file)

Principal Component Analysis (pca)
----------------------------------
  The module can detect several anomalies. It is based on time series analysis
  of traffic volume and entropy values.
  The module splits traffic into normal and residual sub-spaces and then it looks for
  changes in residual subspace.

  Note. The module has long initial phase.

Traffic Repeater (traffic_repeater)
-----------------------------------
  Traffic_repeater is used for re-transmission of incoming libtrap messages from
  input interface to output interface.
  The module can be used as a bridge between two modules (or set of modules) that
  communicate by different interface type.

Transit Filter (transitfilter)
------------------------------
  The module filters incoming UniRec messages. The module has one input interface
  and two output interfaces. Based on configured address prefixes, the incoming
  messages are resent via the first or the second output interface.

TrapDump (trapdump)
-------------------
  TrapDump is used for storage of incoming messages received by libtrap interface.
  Messages are stored into file in trapcap format that can be read by TrapReplay.
  Trapcap format stores payload of messages including internal libtrap message headers
  and buffer headers.

TrapReplay (trapreplay)
-----------------------
  TrapReplay is used as a complement for TrapDump. TrapReplay loads the stored
  messages and resends them via libtrap interface.

DNS Tunnel detection (tunnel_detection_dns)
-------------------------------------------
  This module can detect DNS tunnels and other anomalies such as (D)DOS attacks.
  Module stores statistics about communication of IP addresses. Each time interval,
  the module searches for an anomaly in DNS traffic generated by each IP address.

    Module uses limits and thresholds of several features for detection:
        * count of unique letters in domain,
        * count of numbers in domain,
        * percent of unique access to domain,
        * count of subdomains,
        * size and count of packets from certain IP address,
        * detection of suspected TXT, NS, MX and CNAME record.

  All limits and thresholds can be changed from the default values by a user.
  Module internally uses the implementation of B+ tree data structure for storage
  of IP addresses and the prefix tree for domain names storage. These structures are
  optimized for high performance of the module.
  The module can use input data via libtrap interface or from a PCAP file.

