Sunday, February 15, 2015

ntopng 1.2.2 on Ubuntu 14.04, revisited

I was doing some random research on ntopng a few days ago and I happened to stumble upon a page that was linking to my own blog. I didn't ever think I'd see the day where someone would link to one of my tutorials, especially on a sub-Reddit. This blog was mainly created for myself, little things like this motivate me to post more often.

My ntopng on Ubuntu 14.04 LTS Server post was created only 7 months ago. Unfortunately my tutorial didn't work out for that individual user, and it goes to show how quickly documentation can become inconsistent, especially in the open source world. So I've decided to revisit the topic and redocument it again from scratch; below are directions for installing ntopng 1.2.2 on Ubuntu 14.04.

Installing ntopng

Directions for installing ntopng seem liked they are far more streamlined compared to when I first did this last July. I'll be simply following the official directions for their stable build packages.

Pull down the apt-ntop-stable.deb package using wget, and install with dpkg:

ubuntu@ubuntu-14-04:~$ sudo -i
[sudo] password for ubuntu: 

root@ubuntu-14-04:~# wget
root@ubuntu-14-04:~# ls

root@ubuntu-14-04:~# dpkg -i apt-ntop-stable.deb 
Selecting previously unselected package apt-ntop-stable.
(Reading database ... 55712 files and directories currently installed.)
Preparing to unpack apt-ntop-stable.deb ...
Unpacking apt-ntop-stable (2.1-288) ...
Setting up apt-ntop-stable (2.1-288) ...
Adding ntop key to apt keyring

The apt-ntop-stable.deb package doesn't install ntopng itself, it's simply files to add the repository. See below:

root@ubuntu-14-04:~# dpkg -l | grep ntop
ii  apt-ntop-stable       2.1-288        all          ntop apt package repository

root@ubuntu-14-04:~# dpkg -L apt-ntop-stable

Run apt-get update to update your system repositories and install the packages as per directions:

root@ubuntu-14-04:~# apt-get update
root@ubuntu-14-04:~# apt-get -y install pfring nprobe ntopng ntopng-data n2disk nbox

Here's a short description of what each package in the family does:

root@ubuntu-14-04:~# dpkg -s pfring nprobe ntopng ntopng-data n2disk nbox | egrep "^Package|^Description"

Package: pfring
Description: PF_RING (

Package: nprobe
Description: A network probe.

Package: ntopng
Description: Web-based traffic monitoring.

Package: ntopng-data
Description: Data files (geoip) for ntopng.

Package: n2disk
Description: A packet-to-disk application.

Package: nbox
Description: Web management interface for ntop apps.
  • pfring is a module that allows for high-speed package captures, it's recommended to enable this if you plan on capturing on high-traffic interfaces.
  • nProbe is simply the NetFlow probe, for example you can setup multiple probes throughout your network and send all the NetFlow data to a central ntopng instance to visualize all the traffic.
  • n2disk allows you to efficiently write huge volumes of packet captures to disk without packet loss.

A lot of packages will be installed, and at the very end you should see the following message:


You can now point your browser to https://localhost/

The default user is nbox with password nbox

  • Don't forget that the address is https and not regular https. If you use http it will direct you to the Apache2 Ubuntu Default Page.

Before you visit the https://localhost page, however, restart the apache2 service:

root@ubuntu-14-04:~# service apache2 restart
 * Restarting web server apache2                                                                                              AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using Set the 'ServerName' directive globally to suppress this message

I was getting a Service Unavailable error, and the restart fixed this.

Starting ntopng with the nBox web GUI

Unlike before where we had to configure things manually, the new ntop UI or nBox web GUI makes many of the configurations trivial.

After logging into https://localhost with the default credentials (nbox:nbox), at the top of the nBox dashboard:

  • Applications > ntopng
  • Under the Configuration > General
  • Select which interfaces you want to monitor. If you want to select multiple, hold <Ctrl> and click.
  • Enable the service to startup automatically, if needed.

You can edit other settings under Hosts, Flows, Directory, and Advanced.

When finished, click Save Changes. Then click back to the Status tab, and click On for the interface you selected.

The interface will tell you that you can now access ntopng at the http://<server IP>:3000 address.

Note that additional changes under the Configuration tab first require you to stop the ntopng service by clicking the Off button for your interface under the Status tab.

Change those default credentials

nBox GUI

On the of the nBox web GUI, System > Users. Web Users > for the already selected nbox user click Change Pwd.

ntopng GUI

The default username and password for the ntopng web interface is admin. To change the defaults, after logging into the web interface, click the Gear Icon > Manage Users, for the admin user, click Manage and change the password.


The nBox web UI greatly simplified configuration of all the components of the ntop family, in addition, it's refreshing to see that the ntopng UI is becoming more and more refined with each release.

Monday, February 9, 2015

PPPoE on Ubuntu 14.04

It took me over the course of a week and probably close to 20 hours of troubleshooting to finally get my Linux router to establish a PPPoE connection to my provider. There were two major hurdles I overcame:

  1. The first was figuring out that the VLAN going to the DSL modem needed to be manually tagged. I found out this out by connecting my Windows laptop and tagging it's VLAN on the network adapter after seeing that it was also tagged in the router they used to setup my initial connection. I suspect they do this to differentiate between the IP phone and cable box traffic.
  2. The second hurdle was properly configuring PPPoE properly in Linux. I emphasize properly because PPPoE in Linux is one of those topics that are barely documented, or if it is documented its done 50 different ways, all which don't work correctly for you — kind of like LDAP.

Adding VLAN support

Note this part may not be needed, double check your settings on a working router to see if the VLAN is configured for the WAN interface. If you are able to receive a DHCP lease, but can't establish a PPPoE connection or see any response in the PPPoE logs, you may need to tag the VLAN on your network interface.

This is the part which probably took up the majority of my time because when I ran the pppoeconf utility (like many tutorials and StackOverflow responses tell you to do), it would simply hang. I would suggest trying to use pppoeconf first to configure PPPoE (there are several tutorials out there), if that doesn't work for you, try the procedures outlined in this tutorial.

Tagging VLANs isn't supported by default in Ubuntu 14.04, luckily, the vlan package can do that for us. Install the the vlan package:

root@ubuntu:~# aptitude -yvV install vlan

Load the 8021q module and verify it is loaded:

root@ubuntu:~# modprobe 8021q
root@ubuntu:~# lsmod | grep ^8021q
8021q                  24712  0 

Ensure this module is loaded each time at boot by appending it to /etc/modules:

root@ubuntu:~# echo "8021q" >> /etc/modules

Use the vconfig utility to add a VLAN to the specified interface:

root@ubuntu:/etc/network# vconfig add eth0 20
Added VLAN with VID == 20 to IF -:eth0:-
  • In the above I tagged my eth0 interface with VLAN 20.

As additional verification you can cat the contents of /proc/net/vlan/config which would give you output similar to the following:

root@ubuntu:/etc/network# cat /proc/net/vlan/config 
VLAN Dev name  | VLAN ID
eth0.20        | 20  | eth0

This will allow you to reference the VLAN-tagged interface as eth0.20. As another example, if you were to tag your eth1 interface with VLAN 50 the interface name would be eth1.50.

Network interfaces

After issuing vconfig, you should be able to create and reference the interface in the /etc/network/interfaces file.

# WAN interface
auto eth0
iface eth0 inet dhcp

# WAN interface tagged with VLAN 20
auto eth0.20
iface eth0.20 inet manual
 vlan-raw-device eth0

Now if you have a working physical connection from your ISP facing interface to the DSL modem, you should be able to get a DHCP lease with an private IP address on the original interface, in this case eth0.

The trick was understanding that I had to tell my PPPoE client to use the eth0.20 interface, not eth0, to send the initial request for the PPPoE connection. This is where the pppoeconf setup would hang for me.


Unfortunately it's very easy to get confused between all the PPP/PPPoE packages, e.g. ppp, pppconfig, pppoe, pppoeconf, rp-pppoe, etc. Even worse, is they all dump their configuration files and scripts in the same place, the /etc/ppp directory.

The client that I ended up using was rp-pppoe by Roaring Penguin Software.

Download the tar archive from their website:

root@ubuntu:~# wget
2015-02-07 17:15:55 (167 KB/s) - ‘rp-pppoe-3.11.tar.gz’ saved [223234/223234]

Install the build-essential package which contains additional utilities needed to build packages from source:

root@ubuntu:~# aptitude -yvV install build-essential

Unarchive, change into the unarchived directory, and run the ./go script:

root@ubuntu:~# tar xvf rp-pppoe-3.11.tar.gz

root@ubuntu:~# cd rp-pppoe-3.11/

root@ubuntu:~/rp-pppoe-3.11# ls
configs  doc  go  go-gui  gui  man  README  rp-pppoe.spec  scripts  SERVPOET  src

root@ubuntu:~/rp-pppoe-3.11# ./go

Now if rp-pppoe compiled correctly it should kick off a script immediately afterwards that prompts you for input:

Welcome to the Roaring Penguin PPPoE client setup.  First, I will run
some checks on your system to make sure the PPPoE client is installed

Looks good!  Now, please enter some information:


>>> Enter your PPPoE user name (default dsluser


>>> Enter the Ethernet interface connected to the DSL modem
For Solaris, this is likely to be something like /dev/hme0.
For Linux, it will be ethn, where 'n' is a number.
(default eth0): eth0.20

Do you want the link to come up on demand, or stay up continuously?
If you want it to come up on demand, enter the idle time in seconds
after which the link should be dropped.  If you want the link to
stay up permanently, enter 'no' (two letters, lower-case.)
NOTE: Demand-activated links do not interact well with dynamic IP
addresses.  You may have some problems with demand-activated links.
>>> Enter the demand value (default no): 


Please enter the IP address of your ISP's primary DNS server.
If your ISP claims that 'the server will provide DNS addresses',
enter 'server' (all lower-case) here.
If you just press enter, I will assume you know what you are
doing and not modify your DNS setup.
>>> Enter the DNS information here: server


>>> Please enter your PPPoE password:    
>>> Please re-enter your PPPoE password: 


Please choose the firewall rules to use.  Note that these rules are
very basic.  You are strongly encouraged to use a more sophisticated
firewall setup; however, these will provide basic security.  If you
are running any servers on your machine, you must choose 'NONE' and
set up firewalling yourself.  Otherwise, the firewall rules will deny
access to all standard servers like Web, e-mail, ftp, etc.  If you
are using SSH, the rules will block outgoing SSH connections which
allocate a privileged source port.

The firewall choices are:
0 - NONE: This script will not set any firewall rules.  You are responsible
          for ensuring the security of your machine.  You are STRONGLY
          recommended to use some kind of firewall rules.
1 - STANDALONE: Appropriate for a basic stand-alone web-surfing workstation
2 - MASQUERADE: Appropriate for a machine acting as an Internet gateway
                for a LAN
>>> Choose a type of firewall (0-2): 0

** Summary of what you entered **

Ethernet Interface: eth0.20
User name:          dsluser
Activate-on-demand: No
DNS addresses:      Supplied by ISP's server
Firewalling:        NONE

>>> Accept these settings and adjust configuration files (y/n)? y
Adjusting /etc/ppp/pppoe.conf
Adjusting /etc/ppp/pap-secrets and /etc/ppp/chap-secrets
  (But first backing it up to /etc/ppp/pap-secrets-bak)
  (But first backing it up to /etc/ppp/chap-secrets-bak)

Congratulations, it should be all set up!

Type 'pppoe-start' to bring up your PPPoE link and 'pppoe-stop' to bring
it down.  Type 'pppoe-status' to see the link status.
  • This script will create the /etc/ppp/pppoe.conf populated with configuration parameters that you provided as input.
  • It will also put the username and password into the /etc/ppp/pap-secrets and /etc/ppp/chap-secrets file for you.

Again, note that I specified my VLAN-tagged interface eth0.20 as my interface connected to the DSL modem, and not eth0.

Now you should be able to run pppoe-start:

root@ubuntu:~/rp-pppoe-3.11# pppoe-start
. Connected!

The plog command will show you logging information from the initiated connection:

root@d54250wyk:~# plog
Feb  7 15:32:08 ubuntu pppd[2134]: Remote message: Login ok
Feb  7 15:32:08 ubuntu pppd[2134]: PAP authentication succeeded
Feb  7 15:32:08 ubuntu pppd[2134]: not replacing existing default route via
Feb  7 15:32:08 ubuntu pppd[2134]: local  IP address
Feb  7 15:32:08 ubuntu pppd[2134]: remote IP address
Feb  7 15:32:08 ubuntu pppd[2134]: primary   DNS address
Feb  7 15:32:08 ubuntu pppd[2134]: secondary DNS address

You should now have a ppp0 interface, use ifconfig or ip addr ls to verify it is there.

In the above output you can see the message not replacing existing default route via This was the default route that was obtained from the original DHCP lease on the eth0 interface. There was a bug in the rp-pppoe utility that regardless of settings in the /etc/ppp/pppoe.conf file, it would not obtain and correctly replace the default route via the ppp0 interface.

To resolve this issue, and correctly obtain the default route upon connect, delete all the files in the /etc/ppp/peers/ directory:

root@ubuntu:~# rm -v /etc/ppp/peers/*
removed ‘/etc/ppp/peers/dsl-provider’
removed ‘/etc/ppp/peers/dsl-provider.dpkg-old’
removed ‘/etc/ppp/peers/provider’

Run pppoe-stop and pppoe-start to connect again:

root@ubuntu:/etc/openvpn# pppoe-stop
Killing pppd (2769)
Killing pppoe-connect (2749)

root@ubuntu:/etc/openvpn# pppoe-start
. Connected!

After your connection is established and verified, your configurations for all your network applications, such as iptables should reference the ppp0 interface as your primary interface.

Friday, February 6, 2015

Configuring GeoIP support for Shorewall on Ubuntu 14.04

If you've run fail2ban on any of your servers, see fail2ban with Shorewall, you'll quickly find out that a majority of the banned IPs will originate from many of the same countries, usually China. One of the techniques I previously used to block out an entire country's network range was to use ipsets. I used custom bash scripts to pull down zone information from and import them into separate ipsets for countries I wanted. There were certain limitations with this solution, however, such as maintaining up to date ipsets across multiple servers, some issues with Shorewall losing the ipsets across reboots or restarts, and the US netblock space being to large to fit into a single ipset.

Shorewall version 4.5.4 introduced the ability to support GeoIP 2-character ISO 3166 country codes. This method is far more efficient and easier to maintain as the GeoIP database holds all the netblocks for all the countries in the world in an offline hashed database.

xt_geoip module installation

The first step is to install the xt_geoip kernel module which allows you to reference two letter country codes (e.g. US, CN, UK, CA, FR, etc.) in iptables or Shorewall.

This module can be found in the xtables-addons-common 2.3-1 package, but do not install this default package to use the xt_geoip module. I probably spent over 4 hours trying to figure out why my virtual machine was crashing every time a rule triggered the use of the xt_geoip module only to find out that there's a verified bug that causes a kernel panic.

Patrick Domack PPA to the rescue

Fortunately, if you read the bug comments, a user by the name of Patrick Domack provided his own PPA with a packaged version of xtables-addons-common version 2.6-1~ppa1 which fixes this bug.

Add the PPA with apt-add-repository ppa:patrickdk/general-lucid:

root@ubuntu:~# add-apt-repository ppa:patrickdk/general-lucid

 Packages used for my personal productions systems where newer versions or special patches are needed.
 More info:
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmp24yt3e8g/secring.gpg' created
gpg: keyring `/tmp/tmp24yt3e8g/pubring.gpg' created
gpg: requesting key 4D79B5B5 from hkp server
gpg: /tmp/tmp24yt3e8g/trustdb.gpg: trustdb created
gpg: key 4D79B5B5: public key "Launchpad PPA for Patrick Domack" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

apt-get update to update the repository and verify the new version of xtables-addons-common is available:

root@ubuntu:~# apt-get update

root@ubuntu:~# apt-cache show xtables-addons-common
Package: xtables-addons-common
Source: xtables-addons
Priority: extra
Section: admin
Installed-Size: 326
Maintainer: Pierre Chifflier 
Architecture: amd64
Version: 2.6-1~ppa1

Install xtables-addons-common and verify the correct version is installed:

root@ubuntu:~# aptitude -yvV install xtables-addons-common

root@ubuntu:~# dpkg -l | grep xtables-addons-common
ii  xtables-addons-common               2.6-1~ppa1                    amd64        Extensions targets and matches for iptables [tools, libs]

Building the GeoIP database for xt_geoip

The package will install two scripts in the /usr/lib/xtables-addons directory:

root@ubuntu:~# ls /usr/lib/xtables-addons/
xt_geoip_build  xt_geoip_dl
  • According to the "xt_geoip_dl simply calls wget on the hardcoded URLs and unpacks the retrieved files into the current directory. Then use xt_geoip_build to transform the CSV into the packed format"

The xt_geoip_dl script uses the unzip utility to unpack the a .csv file, install unzip if it isn't already:

root@ubuntu:~# aptitude -yvV install unzip

Change into /tmp and run /usr/lib/xtables-addons/xt_geoip_dl:

root@ubuntu:/tmp# /usr/lib/xtables-addons/xt_geoip_dl

Note that both iptables and shorewall will look for the GeoIP database in the /usr/share/xt_geoip directory, by default it is not created. Also, the build script requires the libtext-csv-xs-perl module to parse the .csv file.

Create the /usr/share/xt_geoip directory and install the required perl module:

root@ubuntu:/tmp# mkdir /usr/share/xt_geoip
root@ubuntu:/tmp# aptitude -yvV install libtext-csv-xs-perl

Run the build script to create the GeoIP database from the .csv files and place the them in the /usr/share/xt_geoip directory:

root@ubuntu:/tmp# /usr/lib/xtables-addons/xt_geoip_build -D /usr/share/xt_geoip *.csv

Loading the module and configuring Shorewall

Load the kernel module and verify it was loaded:

root@ubuntu:~# modprobe xt_geoip
root@ubuntu:~# lsmod | grep ^xt_geoip
xt_geoip               12775  0 

For verification I'm going to add a rule to reject and log any ping from my virtual machine to any address in the United States. Note, I wouldn't recommend blocking all of U.S. in production, because the IP addresses to Google for other countries are still considered U.S. based. Though whitelisting the U.S. address space may be useful in some environments.

Edit the /etc/shorewall/rules file:

Ping(REJECT):info $FW  net:^[US]
  • The format for referencing countries is ^[<Country Code>].
  • Multiple countries can be specified in the brackets, e.g. ^[US,FR,CA,UK].
  • See for more information.

Check and restart Shorewall:

root@ubuntu:/etc/shorewall# shorewall check
root@ubuntu:/etc/shorewall# shorewall restart

Now we'll test ping to

root@ubuntu:/etc/shorewall# ping -n -c 2
PING ( 56(84) bytes of data.
From icmp_seq=1 Destination Host Unreachable
From icmp_seq=1 Destination Host Unreachable

--- ping statistics ---
0 packets transmitted, 0 received, +2 errors

Working as intended. Let's see what Shorewall logged:

Feb  6 23:34:27 ubuntu kernel: [  942.806263] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9477 DF PROTO=ICMP TYPE=8 CODE=0 ID=5713 SEQ=1 
Feb  6 23:34:27 ubuntu kernel: [  942.807537] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9478 DF PROTO=ICMP TYPE=8 CODE=0 ID=5713 SEQ=1 
Feb  6 23:35:26 ubuntu kernel: [ 1002.101682] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63477 DF PROTO=ICMP TYPE=8 CODE=0 ID=5730 SEQ=1 
Feb  6 23:35:26 ubuntu kernel: [ 1002.101924] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC= DST= LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63478 DF PROTO=ICMP TYPE=8 CODE=0 ID=5730 SEQ=1 

And for more verification, to see it isn't blocking IP addresses that are non-US owned, we'll ping

root@ubuntu:/etc/shorewall# ping -n -c 2
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=51 time=297 ms
64 bytes from icmp_seq=2 ttl=51 time=296 ms

--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 296.200/296.846/297.493/0.845 ms


Using GeoIP support in Shorewall is a much quicker way to blacklist or whitelist large country IP address ranges, when compared to using just ipsets. Though this does not diminish the use cases of ipsets, as they are very useful in many different solutions which I will eventually cover in the future.


Friday, July 25, 2014

fail2ban with Shorewall

When it came to blacklisting attackers trying to brute-force my services, like SSH, my go-to package has always been DenyHosts. However, issues such as recent vulnerabilities and most notably, its removal from the default repositories for Ubuntu 14.04 LTS caused me to finally switch to fail2ban. The biggest advantage fail2ban provides over DenyHosts is that it is more flexible in its actions and types of services it can monitor (DenyHosts only supports services using TCP wrappers).


Install the fail2ban package:

root@localhost:~# aptitude -yvV install fail2ban

After installation, the fail2ban service will automatically be started:

root@localhost:/etc/fail2ban# ps aux | grep [f]ail2ban
root     11232  0.0  1.8 268164  9360 ?        Sl   02:47   0:00 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/

root@localhost:/etc/fail2ban# service fail2ban status
 * Status of authentication failure monitor
 *  fail2ban is running


First, make a copy of the /etc/fail2ban/jail.conf file as /etc/fail2ban/jail.local, as per

Every .conf file can be overridden with a file named .local. The .conf file is read first, then .local, with later settings overriding earlier ones. Thus, a .local file doesn't have to include everything in the corresponding .conf file, only those settings that you wish to override. Modifications should take place in the .local and not in the .conf. This avoids merging problem when upgrading.

root@localhost:/etc/fail2ban# cp -v jail.conf jail.local
‘jail.conf’ -> ‘jail.local’

By default fail2ban uses iptables to control its default banning actions. Luckily using Shorewall is as simple as changing a single line in the /etc/fail2ban/jail.local configuration file.



# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
#banaction = iptables-multiport
banaction = shorewall

The /etc/fail2ban/action.d/shorewall.conf configuration file comments explains this in more detail.

All files in the /etc/fail2ban/action.d/ drop configuration directory control actions to an accompanying service. For example, action.d/ipfw.conf controls how fail2ban tells ipfw, used in FreeBSD systems, what command to run to ban an offending host. Likewise, action.d/shorewall.conf controls what commands are run by shorewall in order to ban an offender.

In /etc/fail2ban/action.d/shorewall.conf the important directives to pay attention to for our setup are actionban, actionunban, and blocktype:

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionban = shorewall <blocktype> <ip>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
actionunban = shorewall allow <ip>


# Option:  blocktype
# Note:    This is what the action does with rules.
#          See man page of shorewall for options that include drop, logdrop, reject, or logreject
# Values:  STRING
#blocktype = reject
blocktype = drop
  • To reiterate, when fail2ban bans a host, the command shorewall drop <offending IP address> will be run.

In the above configuration I changed the blocktype to drop rather than reject. Rather than getting an ICMP destination unreachable response from my server when attackers are brute-forcing my services like SSH after they have been banned, drop will cause their connections to timeout instead.

Also, shorewall needs to be updated to handle the banning action correctly. By default, it will only blacklist hosts part of a new TCP connection, but not already existing or established connections.



Restart the fail2ban and shorewall services:

root@localhost:/etc/fail2ban# service fail2ban restart
 * Restarting authentication failure monitor fail2ban                         [ OK ]

root@localhost:/etc/shorewall# service shorewall restart
Restarting "Shorewall firewall": done.

Seeing it in action

After leaving fail2ban running for a couple hours, you are inevitably going to get some banned hosts. You can see that fail2ban is working correctly by checking the /var/log/fail2ban.log log file. Here's an example of some IPs that were banned:

2014-07-23 19:58:16,068 fail2ban.actions: WARNING [ssh] Ban
2014-07-23 20:08:16,818 fail2ban.actions: WARNING [ssh] Unban
2014-07-23 20:46:02,506 fail2ban.actions: WARNING [ssh] Ban
2014-07-23 20:56:03,249 fail2ban.actions: WARNING [ssh] Unban
  • Note the hosts were unbanned after 10 minutes, which is the default timer.
  • To increase the ban time, change the bantime value in the /etc/fail2ban/jail.local configuration file.

To verify that fail2ban did indeed pass the IPs to shorewall use the shorewall show dynamic command:

root@localhost:/etc/fail2ban# shorewall show dynamic
Shorewall Chain dynamic at localhost - Fri Jul 25 07:21:26 EDT 2014

Counters reset Wed Jul 23 03:07:05 EDT 2014

Chain dynamic (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *           
    0     0 DROP       all  --  *      *  


Sunday, July 13, 2014

Redirecting Shorewall log messages to custom file

By default Shorewall will log firewall messages to /var/log/messages if you are running CentOS, or /var/log/syslog if you are running Ubuntu.

Sample log message from Shorewall using default format:

Jul 13 01:30:42 localhost kernel: [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC= DST= LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 

Despite the LOGFILE=/var/log/messages parameter in the shorewall.conf file, this directive isn't used to redirect logs to a custom file. As described from man shorewall.conf:

    This parameter tells the /sbin/shorewall program where to look for Shorewall messages
    when processing the dump, logwatch, show log, and hits commands. If not assigned or if
    assigned an empty value, /var/log/messages is assumed. For further information, see

Using rsyslog to redirect messages

Configurations done through rsyslog will allow Shorewall log messages to be redirected to a custom file.

Create the /etc/rsyslog.d/40-shorewall.conf file:

# touch /etc/rsyslog.d/40-shorewall.conf
# ls -l /etc/rsyslog.d
total 8
-rw-r--r-- 1 root root  311 Mar 17  2012 20-ufw.conf
-rw-r--r-- 1 root root    0 Jul 13 02:04 40-shorewall.conf
-rw-r--r-- 1 root root 1655 May 19  2013 50-default.conf
  • The numerical prefix determines the order at which the configuration files are read by rsyslog, i.e. settings in 40-shorewall.conf will be configured before 50-default.conf.

Edit /etc/rsyslog.d/40-shorewall.conf:

:msg, contains, "Shorewall:" -/var/log/shorewall.log
& ~
  • This is an rsyslog property-based filter which matches any log messages which contain the string Shorewall: and redirects them to the /var/log/shorewall.log log file.
  • The "& ~" tells rsyslog to stop processing the message after it was written to the log. Without the "& ~", messages would continue to be written to local files such as /var/log/syslog.
    • If you wanted messages to still show up in /var/log/{syslog,messages} then the "& ~" can be omitted.

Anytime custom log files are made, an accompanying /etc/logrotate.d configuration file should be created or updated. Shorewall by default creates /etc/logrotate.d/shorewall which controls rotation of the /var/log/shorewall-init.log file. Add a directive to control /var/log/shorewall.log:

# cat /etc/logrotate.d/shorewall
/var/log/shorewall-init.log {
    rotate 4
    create 0640 root adm

/var/log/shorewall.log {
    rotate 52
    compressext .gz
    create 0640 root adm
  • This will rotate the shorewall.log log file every week and store archived log files for 52 weeks (1 year).

Restart the rsyslog service:

# service rsyslog restart

The restart should have triggered the /var/log/shorewall.log file to be created, if not, try to port-scan or trigger an event and see if the file was created:

# ls -l /var/log/shorewall.log
-rw-r----- 1 syslog adm 0 Jul 13 02:27 /var/log/shorewall.log

rsyslog template

I created a template to shorten the log messages, as Shorewall log messages can be quite long and verbose, by adding another line to /etc/rsyslog.d/40-shorewall.conf:

$template shorewall-template,"%timegenerated% %msg%\n"
:msg, contains, "Shorewall:" -/var/log/shorewall.log;shorewall-template
& ~
  • rsyslog filtering does not follow a simple syntax, so I recommend reading their official documentation to learn exactly how it works, as rsyslog can be leveraged to do very complex and powerful configurations.
  • Though not much, the template removes the localhost kernel: portion in the message:
  • From this:
    Jul 13 01:30:42 localhost kernel: [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC= DST= LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 
    To this:
    Jul 13 01:30:42 [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC= DST= LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 
  • Full list of properties available here.

Wednesday, July 9, 2014

Initial network analysis of stray BitTorrent traffic

In my previous blog post I detailed a basic installation of ntopng. Interestingly, I was able to detect odd traffic destined to one of my desktops in my network ‒ no less than a day after I had setup ntopng on a development server. Network forensics and analysis is a domain I've been meaning to level up, so this was a perfect opportunity quest to explore.

In the ntopng web interface under the Flows section, ntopng had identified an active flow with a duration of over 4h, 14 min originating from the IP address, which resides in Kiev, Ukraine. There was a nice little Ukraine flag by the IP in the web interface, so it quickly caught my eye. In addition, the flow was identified as the BitTorrent protocol.

Example screenshot of Active Flows section inside web interface

Foreign IP addresses are to be expected, especially in regards to the BitTorrent protocol. The problem, however, was that my desktop wasn't running a BitTorrent client at the time and BitTorrent almost always connects to several peers at once for both (sometimes simultaneously) uploads and downloads. So my immediate questions were how was this traffic getting through the router firewall and why was it only this single IP address? Seeing a bunch of foreign flags in the web interface at once would have actually worried me less.

Since the flow was still active, I decided to see what it looked like on the wire with tcpdump by using a BPF to match any traffic to or from that Ukraine host:

root@trustytahr:/var/tmp# tcpdump -v -n -i eth0 not tcp port 22 and ip host
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:14:49.233184 IP (tos 0x0, ttl 107, id 5434, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.356179 IP (tos 0x0, ttl 107, id 5449, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.438760 IP (tos 0x0, ttl 107, id 5460, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.544598 IP (tos 0x0, ttl 107, id 5470, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.583587 IP (tos 0x0, ttl 107, id 5475, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.735788 IP (tos 0x0, ttl 107, id 5489, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.804994 IP (tos 0x0, ttl 107, id 5498, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.839829 IP (tos 0x0, ttl 107, id 5502, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.926611 IP (tos 0x0, ttl 107, id 5512, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
13:14:49.965995 IP (tos 0x0, ttl 107, id 5516, offset 0, flags [none], proto UDP (17), length 129) > UDP, length 101 
  • UDP packets were being sent several times a second by with random source ports.
  • All of these packets were being sent to the same UDP port, 31368, on
  • -v specifies verbose mode, and -n prevents tcpdump from doing a reverse DNS lookup on the IP addresses.

Seeing that the traffic was identified as BitTorrent and destined to, I checked three things on (a desktop running Windows 7):

  1. Find any connections matching port 31369, and also see if there were any interesting or malicious processes using the port.
  2. C:\Windows\system32>netstat -nabo | findstr "31369"
    • No connections found.
  3. Double-check if µTorrent was running. A quick check in the system tray and Windows Task Manager verified it wasn't.
  4. See what random port µTorrent was using; µTorrent assigns itself a random UDP port for incoming connections.

  5. 31389 is indeed the randomly assigned port for incoming connections
    Also take note how Enable UPnP port mapping and Enable NAT-PMP port mapping were enabled

For further exploration, I re-ran tcpdump and included the -w switch which outputs the captured network traffic to a named .pcap file. I then opened it in Wireshark and clicked Follow UDP Stream.

Screenshot of Follow UDP Stream in Wireshark

Searching Google for strings in the output lead me to For Developers section on the website (not to be used with On the page it describes the find_node packets:

Find node is used to find the contact information for a node given its ID. "q" == "find_node" A find_node query has two arguments, "id" containing the node ID of the querying node, and "target" containing the ID of the node sought by the queryer. When a node receives a find_node query, it should respond with a key "nodes" and value of a string containing the compact node info for the target node or the K (8) closest good nodes in its own routing table.

Also, it shows an example packet for a find_node Query, which is the type of packet that I was receiving from the Ukraine host:

find_node Query = {"t":"aa", "y":"q", "q":"find_node", "a": {"id":"abcdefghij0123456789", "target":"mnopqrstuvwxyz123456"}}
bencoded = d1:ad2:id20:abcdefghij01234567896:target20:mnopqrstuvwxyz123456e1:q9:find_node1:t2:aa1:y1:qe

Alternatively, the find_node Response packet looks like so (though my desktop never replied to a query in the time it was an active flow listed in ntopng):

Response = {"t":"aa", "y":"r", "r": {"id":"0123456789abcdefghij", "nodes": "def456..."}}
bencoded = d1:rd2:id20:0123456789abcdefghij5:nodes9:def456...e1:t2:aa1:y1:re

UPnP, NAT-PMP and BitTorrent

UPnP stands for “Universal Plug and Play.” Using UPnP, an application can automatically forward a port on your router, saving you the hassle of forwarding ports manually. We’ll be looking at the reasons people recommend disabling UPnP, so we can get a clear picture of the security risks. —
NAT-PMP allows a computer in a private network (behind a NAT router) to automatically configure the router to allow parties outside the private network to contact it. NAT-PMP runs over UDP port 5351. It essentially automates the process of port forwarding. — Wikipedia

UPnP and NAT-PMP, though different protocols, essentially do the same thing. UPnP or NAT-PMP run as a service on many home-based routers, and is usually enabled by default. BitTorrent often relies on these protocols to allow traffic through networks behind a NAT, which is nearly every home network. All of the major BitTorrent applications support support UPnP and NAT-PMP, which allows these clients to map ports on the router.

Turning off UPnP or NAT-PMP is as simple as going into the router web interface and looking for the checkbox to disable it.


Though the flow eventually stopped after a few hours, before I could test to see if turning off UPnP and NAT-PMP made a difference, I've been using ntopng to continually monitor and analyze other anomalous traffic.

Sunday, July 6, 2014

ntopng on Ubuntu 14.04 LTS Server

Following directions are deprecated, updated instructions are available at ntopng 1.2.2 on Ubuntu 14.04, revisited.

ntopng is the next generation version of the original ntop, a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntopng is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform, MacOSX and on Win32 as well. —

ntopng is different from the original ntop package found in the default Ubuntu repositories. Perhaps the biggest different between the two is that ntopng has an updated and modern HTML5 web interface which has data visualizations that update in real-time, whereas ntop provides a static and very generic looking web interface.

Installing ntopng

Note that packages from the repository below are x64 only. If you are using i386, then ntopng must be built from source.

Following the download pages on and looking for stable Ubuntu packages will eventually lead you to this website repository. The directions below follow those in the listed website.

Add the repository and its accompanying public key:

ubuntu@trustytahr:~$ sudo -i
[sudo] password for ubuntu: 

root@trustytahr:~# /bin/echo -e "deb x64/\ndeb all/" > /etc/apt/sources.list.d/ntop.list

root@trustytahr:~# wget -qO - | sudo apt-key add -
  • Switch completely to the root user, rather than using sudo to run the above commands.

Update apt repositories with apt-get update and ignore the conflicting messages at the end:

root@trustytahr:~# apt-get update

Reading package lists... Done
W: Conflicting distribution: x64/ Release (expected x64 but got )
W: Conflicting distribution: all/ Release (expected all but got )

Install the ntopng and ntopng-data packages:

root@trustytahr:~# apt-get -y install ntopng ntopng-data

The installation creates startup script /etc/init.d/ntopng, and the ntopng binary itself as /usr/local/bin/ntopng. The rest of the relevant files can be all found under the /usr/local/share/ntopng directory. See output of dpkg -L ntopng for a full listing.

Running ntopng

Starting ntopng is as simple as executing the /usr/local/bin/ntopng binary:

root@trustytahr:/usr/local/bin# ./ntopng 
06/Jul/2014 01:15:34 [Ntop.cpp:555] Setting local networks to,,,,,
06/Jul/2014 01:15:34 [Redis.cpp:50] Successfully connected to Redis
06/Jul/2014 01:15:34 [PcapInterface.cpp:81] Reading packets from interface eth0...
06/Jul/2014 01:15:34 [Ntop.cpp:662] Registered interface eth0 [id: 0]
06/Jul/2014 01:15:34 [PcapInterface.cpp:81] Reading packets from interface lo...
06/Jul/2014 01:15:34 [Ntop.cpp:662] Registered interface lo [id: 1]
06/Jul/2014 01:15:34 [Utils.cpp:251] User changed to nobody
06/Jul/2014 01:15:34 [main.cpp:152] PID stored in file /var/tmp/
06/Jul/2014 01:15:34 [HTTPserver.cpp:351] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
06/Jul/2014 01:15:34 [HTTPserver.cpp:352] Please read README.SSL if you want to enable SSL
06/Jul/2014 01:15:34 [HTTPserver.cpp:389] Web server dirs [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts]
06/Jul/2014 01:15:34 [HTTPserver.cpp:392] HTTP server listening on port 3000
06/Jul/2014 01:15:34 [main.cpp:186] Using RRD version 1.4.7
06/Jul/2014 01:15:34 [main.cpp:202] Working directory: /var/tmp/ntopng
06/Jul/2014 01:15:34 [main.cpp:204] Scripts/HTML pages directory: /usr/local/share/ntopng
06/Jul/2014 01:15:34 [Ntop.cpp:181] Welcome to ntopng x86_64 v.1.1.4 (r7806) - (C) 1998-14
06/Jul/2014 01:15:34 [PeriodicActivities.cpp:53] Started periodic activities loop...
06/Jul/2014 01:15:34 [NetworkInterface.cpp:770] Started packet polling on interface eth0...
06/Jul/2014 01:15:34 [NetworkInterface.cpp:770] Started packet polling on interface lo...
  • Above output shows ./ntopng run as a foreground process, optionally it could have been ran as a background process with ./ntopng &.
  • By default the working directory is /var/tmp and the listening port for the web interface is at port 3000.

While the process is running, the web interface should now be accessible via browser at http://<SERVER IP>:3000 — http://localhost:3000 if viewing the web page locally.

The default username is admin and password is admin.

Running ntopng as a service

Running ntopng as a service, rather than a foreground or background script requires additional steps.

Note that ntopng has several options that can be passed at either the command-line or via configuration file. All the options are listed in the man page.

A configuration file should be created at /etc/ntopng/ntopng.conf, below is a configuration file I used (the commented descriptions are excerpts directly taken from man ntopng):

# /etc/ntopng/ntopng.conf
#        The  configuration  file is similar to the command line, with the exception that an equal
#        sign '=' must be used between key and value. Example:  -i=p1p2  or  --interface=p1p2  For
#        options with no value (e.g. -v) the equal is also necessary. Example: "-v=" must be used.
#       -G|--pid-path
#        Specifies the path where the PID (process ID) is saved.
#       -e|--daemon
#        This  parameter  causes ntop to become a daemon, i.e. a task which runs in the background
#        without connection to a specific terminal. To use ntop other than as a casual  monitoring
#        tool, you probably will want to use this option.
#       -i|--interface
#        Specifies  the  network  interface or collector endpoint to be used by ntopng for network
#        monitoring. On Unix you can specify both the interface name  (e.g.  lo)  or  the  numeric
#        interface id as shown by ntopng -h. On Windows you must use the interface number instead.
#        Note that you can specify -i multiple times in order to instruct ntopng to create  multi‐
#        ple interfaces.
#       -w|--http-port
#        Sets the HTTP port of the embedded web server.
#       -m|--local-networks
#        ntopng determines the ip addresses and netmasks for each active interface. Any traffic on
#        those  networks  is considered local. This parameter allows the user to define additional
#        networks and subnetworks whose traffic is also considered local in  ntopng  reports.  All
#        other hosts are considered remote. If not specified the default is set to
#        Commas  separate  multiple  network  values.  Both netmask and CIDR notation may be used,
#        even mixed together, for instance ",".
#       -n|--dns-mode
#        Sets the DNS address resolution mode: 0 - Decode DNS responses  and  resolve  only  local
#        (-m)  numeric  IPs  1  -  Decode DNS responses and resolve all numeric IPs 2 - Decode DNS
#        responses and don't resolve numeric IPs 3 - Don't decode DNS responses and don't  resolve
#       -S|--sticky-hosts
#        ntopng  periodically purges idle hosts. With this option you can modify this behaviour by
#        telling ntopng not to purge the hosts specified by -S. This parameter requires  an  argu‐
#        ment  that  can  be  "all"  (Keep  all hosts in memory), "local" (Keep only local hosts),
#        "remote" (Keep only remote hosts), "none" (Flush hosts when idle).
#       -d|--data-dir
#        Specifies the data directory (it must be writable). Default directory is ./data
#       -q|--disable-autologout
#        Disable web interface logout for inactivity.

Create the /etc/ntopng/ntopng.start as an empty file:

root@trustytahr:~# touch /etc/ntopng/ntopng.start

root@trustytahr:~# ls -l /etc/ntopng/ntopng.start
-rw-r--r-- 1 root root 0 Jul  6 01:41 /etc/ntopng/ntopng.start
  • For some reason the start_ntopng() function in the /etc/init.d/ntopng startup script requires this file in order for the service to start.

Start the ntopng service:

root@trustytahr:~# service ntopng start
Starting ntopng

The ntopng process can be verified with the service ntopng status or in ps aux output:

root@trustytahr:~# ps aux | grep [n]topng
nobody    2343  0.6  1.3 764432 27900 ?        Ssl  01:45   0:00 /usr/local/bin/ntopng /etc/ntopng/ntopng.conf

root@trustytahr:~# service ntopng status
ntopng running as 2343

Changing default password for web interface

Command-line Method

Verify connectivity to redis-server with redis-cli ping command:

root@trustytahr:~# redis-cli ping

Generate an md5 hash of our new password with md5sum:

root@trustytahr:~# echo -n "NewPassword_4ntopng" | md5sum
a13a83f9a23740e9ba453b2df3c41f9a  -

Use this hash in the redis-cli SET user.admin.password command:

root@trustytahr:~# redis-cli SET user.admin.password a13a83f9a23740e9ba453b2df3c41f9a

Web Interface Method

After logging into the web interface:

Settings (Gear Icon) > Manage Users > Set Password


  • If the service fails to start, run /usr/local/bin/ntopng /etc/ntopng/ntopng.conf directly to see all messages on stdout.
  • The log file can be found at: /var/tmp/ntopng/ntopng.log
  • ntopng requires the redis-server service to be running:
  • root@trustytahr:~# service redis-server status
    redis-server is running
    root@trustytahr:~# netstat -tunlp | grep -i redis-server
    tcp        0      0*               LISTEN      2139/redis-server 1

Bug Fixes

The /etc/init.d/ntopng startup script has a few minor bugs that could use a couple fixes:

  • /usr/local/bin/ntopng can actually be run without the root user; however, the ntopng process will not be able to enter the listening interface into promiscuous mode and will still return a successful exit status.
  • The startup script parses for PID_FILE using grep to find the -G switch, but doesn't alternatively search for its synonymous long switch --pid-file.
  • The service ntopng status command will return nothing if ntopng hasn't started.

diff -u output for the above fixes:

root@trustytahr:~# diff -u /etc/init.d/ntopng.orig /etc/init.d/ntopng
--- /etc/init.d/ntopng.orig 2014-07-06 02:30:21.133552653 -1000
+++ /etc/init.d/ntopng 2014-07-06 02:44:55.057578340 -1000
@@ -13,6 +13,11 @@
 # Short-Description: Start/stop ntopng web
+if [ $(id -u) -ne 0 ]; then
+    echo "ntopng requires root."
+    exit 1
 start_ntopng() {
@@ -34,7 +39,7 @@
     if [ -f /etc/ntopng/ntopng.start ] || [ $FORCE -eq 1 ]; then
  echo "Starting ntopng"
- PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+ PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
         if [ -f $PID_FILE ]; then
      PID=$(cat $PID_FILE)
             if [ $PID -gt 0 ]; then
@@ -60,7 +65,7 @@
     if [ -f /etc/ntopng/ntopng.conf ]; then
- PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+ PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
  if [ -f "$PID_FILE" ]; then
      PID=$(cat $PID_FILE)
      if [ $PID -gt 0 ]; then
@@ -86,7 +91,7 @@
  return 0
-    PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+    PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
     if [ -f $PID_FILE ]; then
  PID=$(cat $PID_FILE)
  if [ $PID -gt 0 ]; then
@@ -94,6 +99,8 @@
      echo "No running ntopng pid [$PID] in [$PID_FILE]"
+    else
+        echo "ntopng not running: process id file $PID_FILE not found."
     return 0