One of the greatest strengths of ntopng is its ability to correlate data originating at different layers and at multiple sources together. For example, ntopng can look at IP packets, Ethernet frames and, at the same time, poll SNMP devices. This enables ntopng to effectively perform correlations and observe:

• The behavior of IP addresses (e.g., Is this IP known to be blacklisted?)
• The MAC addresses carrying IP traffic around in the network
• The physical location of the MAC addresses (i.e., physical switches traversed by a given MAC address along with trunk and access ports)

ntopng, starting from version 4.1, capitalizes on this information to implement attack mitigation via SNMP. In other words, ntopng:

1. Uses an indication of compromise known as score to determine whether an IP is an attacker (client score) or a victim (server score).
2. Finds physical switches and access ports where attackers are connected to.
3. Uses SNMP to turn access ports down, thus effectively disconnecting the attackers from the healthy network.

Attack mitigation via SNMP is implemented as an ntopng plugin available in versions Enterprise M and above, and can be enabled from the user scripts configuration page

Let’s see what happens in practice. For this example, an attacker host 192.168.2.149 is configured to run a port scan (nmap -sS) towards 192.168.2.222. ntopng, using traffic and SNMP data is able to identify host 192.168.2.149 as a PcEngines connected to interface gigabitethernet15 of switch 192.168.2.168.

The port scan is immediately detected by ntopng

Indeed, there are many alerted “TCP Connection Refused” flows having 192.168.2.149 as source – apu is the DNS name of 192.168.2.149. Due to this suspicious activity, there is a significant increase in the score of 192.168.2.149

This score is high enough to ensure the attack mitigation via SNMP kicks in. Within a minute from the increase in the host score, mitigation causes the port on the SNMP device to be turned down

From this point on, attacker host 192.168.2.149 is effectively disconnected from the network and, thus, it becomes harmless. It is up to the network administrator now to intervene and do any necessary cleanup operation on the attacker host. Once the issues have been resolved, the SNMP port can be turned up again from the preferences.

Having attack mitigation via SNMP implemented in ntopng is just a preliminary step towards making ntopng not just a monitoring and visualisation tool but also something which proactively prevents attackers from harming the network.

What’s next is the ability to protect a network by mitigating external attackers. nScrub is a good example of a tool that can mitigate DDoS attacks. Stay tuned for news !

Monitoring Industrial IoT and SCADA traffic can be challenging as most open source monitoring tools are designed for Internet protocols. As this is becoming a hot topic with companies automating production lines, we have decided to enhance ntop tools to provide our user community traffic visibility even in industrial environments. This has required to enhance nDPI to detect these protocols and enhance ntopng, our monitoring console, to visualize this traffic by providing enhanced protocol dissection on top of which alerts can be triggered.

To date, nDPI supports modbus, DNP3 and IEC60870 protocols. In particular IEC 60870 is very important as it can be used to detect issues such as

• Unknown telemetry addresses
• Connection loss and restore
• Loss of data coming from remote systems

The standard is quite complex and if you want to monitor this traffic to trigger alerts using open source software your choice is limited to custom scripts for the suricata IDS or Zeek/Malcom. As ntopng has the ability to trigger alerts by means of user-scripts when specific events happen, we have decided to enhance ntopng to dissect this traffic so that it is possible to emit custom alerts when specific communications are detected. In Scada in fact companies usually monitor traffic passively instead of actively dropping specific communications when something goes wrong: this is because the risk to drop a wrong packet is too high compared to the benefit and it is much better to trigger and alert and handle it rather than take the risk.

ntopng has been extended to continuously (i.e. not just the first few packets of a communication) monitor IEC 60870 communications and dissect individual PDUs. This way users can trigger alerts by means of ntopng user scripts. The flexibility introduced in ntopng 4.1.x that scripts can be bound to host pools, allow custom script configurations to be created for specific devices so that each device family has (potentially) its custom ruleset.

The above picture shows how a IEC 60870 is detected and reported by ntopng that in addition to usual latency, throughput, retransmissions… metrics it complements it with specific protocol information that can be used to detect anomalies and trigger alerts.

ntop acknowledges swtich.ch for introducing and assisting us while developing IoT/Scada extensions.

Happy IoT and Scada monitoring!

In the latest ntopng 4.1.x versions (and soon 4.2) we have completely reworked the way alerts are delivered to subscribers. Up to 4.0 the ntopng engine was configured in a single way for all alerts: go to the preferences page and specify where to deliver alerts. This was suboptimal for many reasons, including the fact that it was not possible to send alerts to different recipients on different channels, or selectively decide when send alerts out.

For this reason we have introduced the concept of

• Endpoints
  server account configuration where to send alerts to. It is used to configure once the server parameters (for email for instance you need to configure the server IP, username and password) that you can reuse multiple times.
• Recipient
  Endpoint users to which deliver an alert. For instance once you have configured a email server account endpoint, you can define several recipients that can be reached over the same endpoint simply inheriting the configuration of the endpoint and defining just the recipient email address.

For impatients we have prepared a video that shows you all steps to follow in detail

otherwise you can follow this short tutorial below.

How to Configure Recipients and Endpoints

This is done by selecting the System interface, using the Notification submenu.

Endpoints have to be defined first as follows

Note that there are several endpoint families including

• Email
• ElasticSearch
• Slack
• WebHook
• Discord
• Syslog

At this point you can define a recipient that is the one who will receive the alert message.

Note that you can specify what alert severity and category can be delivered to this person. This way you can for instance deliver security events to one recipient, and network events to another.

Of course you can define multiple recipient and endpoints.

Binding Recipients to Alerts

Now that we have defined where alerts should be delivered we need to specify how/when alerts are delivered to the specified recipients. This is implements through Pools that you can access under the System menu.

Pools are a way to cluster up resources for which we want to apply a specific setup. As you can see from the picture below there are various pool families:

• Hosts
• Interfaces
• Local networks
• SNMP Devices
• Active Monitoring
• Host Pools (pools of host pools)
• Flows
• Devices
• System (Interface)

Suppose you want to send an alert when active monitoring has an alert to report. All you need to do is

• Select the Active Monitoring Tab
• Click on the Edit button and specify on the dropdown menu the recipient we have just defined and save it

If you want to double check if this setup is correct, you can go to an active monitoring resource you defined and edit it.

As you can see on the highlighted text above, the new recipient we defined has been defined and is now in use.

What if now you want do define different recipients for each monitored host? Well you need to go back to the pools page, select active monitoring (this will applies to other tabs) and define new pools as show below, and associate different recipients to them.

Then you can now go back to the active monitoring page and select for each host, the pool you like for such host as shown below

To make things more complicated, you need to master how host pools are defined. Contrary to active monitoring, host pools can be quite complex as you might want to define pool members based on IP addresses, networks (CIDR) and Mac Addresses (great for DHCP networks where IPs are floating).

Final Words

We hope that the concept of recipients and endpoints is now clear. What you can do now is have the flexibility to deliver events to selected recipients in a simple yet effective way. All these features are part of ntopng 4.1.x and soon of the next 4.2 stable release.

Enjoy!

With the introduction of ntopng endpoints and recipients, it is now possible to handle alerts in a flexible fashion by means of recipients. ntopng embeds a SQLite database for turn-key alert storage and reporting. However in large organizations with many alerts scalability of this solution is limited due to the limited number of records (16k) that can be handled. In the latest ntopng 4.1.x versions it is now possible to export alerts in an external ElasticSearch database (not available in the community edition). This post shows you how to use this integration in ntopng 4.1.x and soon 4.2.

As shown in the video the first element to create is an endpoint for ElasticSearch that points to the instance running on our datacenter or on the same host where ntopng is running.

At this point you need to define a recipient for this Endpoint

In order to instruct ntopng to send notifications are sent to this recipient you need to configure the pools that use this recipient.

This is done by clicking on the icon highlighted by the arrow in the above picture that will bing you to the pools page.

For each entity (Hosts, Flows, SNMP…) for which you want to deliver alerts, you need to click on edit and specify on the dropdown menu the list of recipients to which notifications will be delivered. Note that you have always a built-in SQLite recipient enabled and used by ntopng to display alerts in the web GUI.

If you want to check if the notification delivery is working you can check (see the picture below) if the number of uses increases.

At this points alerts are stored in Elastic and they can visualised and explored using Kibana. In order to do that you first need to create an index pattern (menu “Stack Management” -> “Index Patterns”) selecting @timestamp as index as shown in the picture below.

Done this you can visualize alerts and create beautiful dashboards with them

Enjoy !

In ntopng you can write plugins to extend it with custom features. This short tutorial explains you how to do that step-by-step. Here we drive you through the creation of a plugin for generating alerts when an unexpected DNS server is observed: this is useful to spot hosts that have a custom DNS configured or scanners applications.

The plugin source code described in this post can be found here and is has been authored by Daniele Zulberti and Luca Argentieri. These are the steps to implement the plugin.

Step 1: Create the plugin folder

ntopng stores plugins under <installation dir>/scripts/plugins, plugins are grouped in categories, in this case alerts/security is the correct one. So let’s create our folder unexpected_dns, this folder contains all the plugin’s sources and configurations.

$ cd <installation dir>/scripts/plugins/alerts/security
$ mkdir unexpected_dns

Step 2: Create the manifest.lua

manifest.lua contains basic plugin information such as name, description and version. It must return these information with a Lua table:

return {
    title = "Unexpected DNS",
    description = "Trigger an alert when an unexpected DNS server is detected",
    author = "Daniele Zulberti, Luca Argentieri",
    dependencies = {},
}

Table keys are:

• title: The title of the plugin. This is used within the ntopng web GUI to identify and configure the plugin.
• description: The description of the plugin. This is used within the ntopng web GUI to provide a plugin description to the user.
• author: A string indicating the name of the author of the plugin.
• dependencies: A Lua array of strings indicating which other plugins this – plugin depends on. The array can be empty when the plugin has no dependencies.

Now, in the web GUI, under Developer/Plugins click on Reload Plugins button. If it’s all ok you should see Unexpected DNS in the plugin list.

Plugins page:

Unexpected DNS plugin raw:

Step 3: Plugin logic

The main part of the plugin it’s under user_scripts. Every plugins can run scripts for various traffic elements. This script need to analyse flows, so you must tell to ntopng that by creating user_script/flow folder:

$ mkdir -p user_scripts/flow

Now we can start to write some code.

Move in user_script/flow and create unexpected_dns.lua, the file name must be the same as the plugin root directory.

local script = {
   -- Script category
   category = , 
   -- Priority
   prio = ,
   -- NOTE: hooks defined below
   hooks = {},
   -- use this plugin only with this protocol
   l7_proto_id = ,
   -- Specify the default value when clicking on the "Reset Default" button
   default_value = {
      items = {},
   },
   gui = {
   }
}

return script

The script variable contains the configuration of the script. It’s a Lua table where category, priority, hooks, l7_proto_id, default_value and gui, are some of the script table keys.

• • category: specify the category of the scripts, our is under security, so you should set the value to user_scripts.script_categories.security (note that the root directory of the plugin is under alerts/security).
  • priority: Is a number representing the script execution priority, the default priority is 0, lower number have less priority.
  • hooks: a Lua table with hook names as key and callbacks as values. User Script Hooks are events or points in time. ntopng uses hooks to know when to call a user script. A user script defining a hook will get the hook callback called by ntopng. User scripts must register to at least one hook. For our purpose protocolDetected it’s enough, the script runs only when the DNS protocol is detected in a flow. The hooks are defined below the script table definition:

    
    local script = { ... }
    
    -- #################################################
    
    function script.hooks.protocolDetected(now, conf)
       io.write("DNS Protocol Detected\n")
    end

• l7_proto_id: only execute the script for flows matching the L7 application protocol. Set it to 5, the DNS ID in nDPI.
• default_value: the default value for the script configuration. See User Scripts GUI. There is no default configuration for DNS servers, you can set it later in the web GUI configuration page of the script.
• gui: a Lua table specifying user script name, description and configuration. Data is used by ntopng to show the user script configurable from the [User Scripts GUI](https://www.ntop.org/guides/ntopng/plugins/user_script_gui.html#web-gui). Set it as follow:
  lua
gui = {
i18n_title = "unexpected_dns.unexpected_dns_title",
i18n_description = "unexpected_dns.unexpected_dns_description",input_builder = "items_list",
item_list_type = "string",
input_title = i18n("unexpected_dns.title"),
input_description = i18n("unexpected_dns.description"),
}
• unexpected_dns.unexpected_dns_title, unexpected_dns.unexpected_dns_description, unexpected_dns.title, unexpected_dns.description are localisation keys. if the variable name have i18n suffix like i18n_title ntopng automatically converts that localisation keys to the current user language otherwise you can force it using the i18n() function. The localisation keys are stored under <plugin root directory>/locales, and every language have the own file, for example the english language have the en.lua file. See Localization.

Create the locales directory:

$ cd ../..
$ mkdir locales

and the en.lua:

return {
   unexpected_dns_description = "Trigger an alert when not allowed DNS server is detected",
   unexpected_dns_title = "Unexpected DNS",
   title = "Allowed DNS",
   description = "Comma separated values of allowed DNS IPs. Example: 8.8.8.8,8.8.4.4,1.1.1.1",
   status_unexpected_dns_description = "Unexpected DNS server found:",
   alert_unexpected_dns_title = "Unexpected DNS found"
}

Now you can test it reloading the plugin, and executing dig google.com @1.1.1.1 in the shell, dig is a useful tool for interrogating DNS name servers.

$ dig google.com @1.1.1.1

; <<>> DiG 9.16.1-Ubuntu <<>> google.com @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 33360
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;google.com.            IN  A

;; ANSWER SECTION:
google.com.     168 IN  A   216.58.209.46

;; Query time: 23 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: mer ott 14 12:23:52 CEST 2020
;; MSG SIZE  rcvd: 55

Then take a look to the shell where you’ve launched ntopng, you should see a line saying DNS Protocol Detected.

Obviously write a line in the shell is not our goal. We want the script to trigger an alert when a not allowed DNS request is detected, to do so we need to add alert and status definitions, and then change the hook code.

Step 4: Define alert definition

A plugin enables alerts to be generated. All the alerts a plugin is willing to generate require a file in plugin subdirectory <plugin root directory>/alert_definitions/. The file contains all the necessary information which is required to properly show, localise and format an alert. The name of the file has this format: alert_plugin_name.lua, in our case: alert_unexpected_dns.lua

The file must return a Lua table with the following keys:

• alert_key: A constant uniquely identifying this alert.
• i18n_title: A string indicating the title of the alert.
• i18n_description (optional): Either a string with the alert description or a function returning an alert description string.
• icon: A Font Awesome 5 icon shown next to the i18n_title.
• creator (optional): this function creates the alert and prepares a JSON.

The alert_key is a constant uniquely identifying the alert. Constants are available in file <installation dir>/scripts/lua/modules/alert_keys.lua. The file contains a table alert_keys with two sub-tables:

• ntopng
• user

Plugins distributed with ntopng must have their alert_keys defined in sub-table ntopng. User plugins must have their alert_keys defined in sub-table user.

Sub-tables can be extended adding new alert_key s to either the ntopng or the user table. Each alert_key has an integer number assigned which must be unique.

Alert description i18n_description can be either a string with the alert description or a function returning an alert description string. See Alert Definitions.

alert_unexpected_dns.lua:

local alert_keys = require ("alert_keys")

-- #################################################

local function createUnexpectedDNS(alert_severity, dns_info)
    local built = {
        alert_severity = alert_severity,
        alert_type_params = dns_info 
    }

    return built
end

-- #################################################

return {
    alert_key = alert_keys.ntopng.alert_unexpected_dns_server,
    i18n_title = "unexpected_dns.alert_unexpected_dns_title",
    icon = "fas fa-exclamation",
    creator = createUnexpectedDNS,
}

The createUnexpectedDNS function has many parameters as you need. Our are:

• alert_severity: tells ntopng the severtity of the alert, one of {info, warning, error}. We set it in the status definition.
• alert_type_params: the value will go in the JSON file.

Step 5: Define status definition

A plugin enables one or more statuses to be set on certain flows. A flow can have multiple statuses set and statuses can be associated to alerts. Flow statuses must be defined in plugin sub-directory <plugin root directory>/status_definitions/ and they are set calling flow.triggerStatus. In the next step you can see the final code for unexpected_dns.lua. Definition is done using a Lua files, one file per status. The name of the file has this format: status_plugin_name.lua, in our case: status_unexpected_dns.lua

A flow status definition file must return a Lua table with the following keys:

• status_key: A constant uniquely identifying this status.
• i18n_title: A string indicating the title of the status.
• i18n_description (optional): Either a string with the flow status description or a function returning a flow status description string.
• alert_type (optional): When an alert is associated to the flow status, this key must be present. Key has the structure alert_consts.alert_types.<an alert key>, where <an alert key> is the name of a file created in Alert Definitions, without the .lua suffix.
• alert_severity (optional): When an alert is associated to the flow status, this key indicates the severity of the alert. Key has the structure alert_consts.alert_severities.<alert severity>, where <alert severity> is one among the available alert severities.

status_unexpected_dns.lua:

local alert_consts = require("alert_consts")
local status_keys = require ("flow_keys")

return {
    status_key = status_keys.ntopng.status_unexpected_dns_server,
    alert_severity = alert_consts.alert_severities.error,
    alert_type = alert_consts.alert_types.alert_unexpected_dns,
    i18n_title = "unexpected_dns.unexpected_dns_title",
    i18n_description = "unexpected_dns.status_unexpected_dns_description",
}

At the end this should be the plugin directory tree :

unexpected_dns/
├── alert_definitions
│   └── alert_unexpected_dns.lua
├── locales
│   └── en.lua
├── status_definitions
│   └── status_unexpected_dns.lua
├── user_scripts
│   └── flow
│       └── unexpected_dns.lua
└── manifest.lua

Step 6: Change and finish the script file

Now we can modify the hook in unexpected_dns.lua to detect the not allowed DNS servers.

function script.hooks.protocolDetected(now, conf)
   if(table.len(conf.items) > 0) then
      local ok = 0
      local flow_info = flow.getInfo()
      local server_ip = flow_info["srv.ip"]

      for _, dns_ip in pairs(conf.items) do
         if server_ip == dns_ip then
            ok = 1
            break
         end
      end

      if ok == 0 then
         flow.triggerStatus(
            flow_consts.status_types.status_unexpected_dns.create(
               flow_consts.status_types.status_unexpected_dns.alert_severity,
               server_ip
            ),
            100, -- flow_score
            0, -- cli_score
            100 --srv_score
         )
      end
   end
end

The parameters of the hook are:

• now: An integer indicating the current epoch.
• conf: A table containing the user script configuration submitted by the user from the User Scripts GUI. Table can be empty if the script doesn’t require user-submitted configuration.

The if statement checks if there are allowed DNS server set by the user in the GUI. With flow.getInfo() we can retrieve the server ip by accessing the Lua table that the function returns.

We can simply scan the table and compare the value with the server ip. We can break as soon as a match is found. If there aren’t any match we call flow.triggerStatus with this parameters:

• flow_consts.status_types.status_unexpected_dns.create: this is the createUnexpectedDNS function defined in the alert definition, so we pass the alert severity and the server ip.
• flow_score, cli_score, srv_score: score goes from 0 to 100 and represent the relevance of the flow, the client and the server.

Test the final script

Now the plugin is ready to run. Reload the plugins and configure a list of DNS to test it:

• go under Settings/User Scripts/
• click on the Flows tab
• click on the edit button.

• search the plugin in All tab by using the search box in the upper right corner.
• click on edit

• Insert the IP list in CSV format in the input box.
• click Apply

Now launch the dig command as done before. You shold see a flow alert in the top bar near the local devices icon.

In the info of the alerted flow you can find the Flow Alerted raw saying: Unexpected DNS server found: [Score: 100].

To check if an alert was triggered follow this steps:

• go under Alerts/Detected Alerts/
• click the Flow Alerts tab

you should see a line with the unexpected DNS alert description.

Final Words

The plugin is complete and working. See the Documentation for further information about ntopng and the plugins development. It is now time for you to write a plugin and contribute to the ntopng development.

We are pleased to introduce ntopng 4.2 that introduces several new features and breakthroughs while consolidating the changes introduced with 4.0. The main goals of this release include

• Enhance and simplify how alerts are delivered to consumers
• Many internal components of ntopng have been rewritten in order to improve the overall ntopng performance, reduce system load, and capable of processing more data while reducing memory usage with respect to 4.0.
• Cybersecurity extensions have been greatly enhanced by leveraging on the latest nDPI enhancements that enabled the creation of several user scripts able to supervise many security aspects of modern systems.
• Behavioral traffic analysis and lateral traffic movement detection for finding cybersecurity threats in traffic noise.
• Initial Scada support with native IEC 60870-5-104 support. We acknowledge switch.ch for having supported this development.
• Consolidation of Suricata and external alerts integration to further open ntopng to the integration of commercial security devices.
• SNMP support has been enhanced in terms of speed, SNMPv3 protocol support, and variety of supported devices.
• New REST API that enabled the integration of ntopng with third party applications such as CheckMK.

As the list of new features and enhancements we plan to write new posts, make videos and organise online training sessions to introduce our community to this new release.

Flexible Alerts Handling

The way alerts are delivered to interested recipients has been completely reworked. Before version 4.2, all the generated alerts were delivered to all recipients, causing issues such as:

• Recipients flooded with too many alerts
• Recipients getting alerts they’re not interested into

For these reasons, we wanted to rethink and redesign the way alerts were delivered to recipients as described in the user’s guide We wanted to obtain enough flexibility to:

• Avoid flooding recipients with too unwanted alerts by introducing flexible alerts delivery.
• Selectively send alerts to a recipient subset based on:
  • Severity-based criteria (e.g., only send alerts with severity error or higher to that particular recipient)
  • Type-based criteria (e.g., only send security-related alerts to that particular recipient)

For the sake of example, the way alerts are now delivered to recipients, allows you to create policies such as

• Send security-related alerts to an Elasticsearch instance managed by the SecOps
• Send network-related alerts via email to the NetOps
• Send ntopng login attempts and configuration changes on the Discord channel of the DevOps
• Send alerts with severity error or higher to SecOps, NetOps, and DevOps together

See this post for a comprehensive discussion and additional examples.

Scalable SNMP v2c/v3 support

This 4.2 release also carries an almost-completely rewritten SNMP engine. The new engine

• Supports SNMP v2c and v3
• Features SNMP bulk requests to greatly improve speed
• Polls multiple devices in parallel to increase throughput

This is a great step forward compared to the SNMP engine featured in version 4.0 which was definitely slower.

With the new engine is also possible to enforce SNMP attack mitigation to toggle the administrative status of an SNMP port to down, when a malicious host is connected to it.

Additional New Features

Among the new features shipped with version 4.2 it is worth mentioning

• Traffic Behavioral Analysis
  • Periodic Traffic
  • Lateral Movements
  • TLS with self-signed certificates, issuerDN, subjectDN
• Support for Industrial IOT and Scada with modbus, DNP3 and IEC60870
• Active monitoring
  • Support for ICMP v4/v6, HTTP, HTTPS and native Speedtest for measuring the available bandwidth.
  • Ability to generate alerts upon unreachable, or slow hosts or services.
• Detection of unexpected servers.
• DHCP, NTP, SMTP, DNS.
• Services map.
• Enhance nIndex integration to maximize flows dump performance and provide better flow drill-down features.
• MacOS package.

For a comprehensive list of features, changes, and fixes, have a look at the CHANGELOG.

So now it’s time for you to give version 4.2 a try! And feel free to join the discussion!

Telegram is a popular messaging application that many people use daily to do instant messaging and receive notifications. As of ntopng 4.2, it is now possible to deliver alerts to external entities including Slack, email and Discord.

This post will show you how the Telegram alert endpoint has been developed so that readers can learn how to contribute to the ntopng development by coding new integrations. For a complete guide about alert endpoints, please refer to the ntopng user’s guide, whereas the complete telegram endpoint source code can be found here.

We suppose that you have downloaded the ntopng code from github. Done that, go to ntopng/scripts/plugins/endpoints: here you can find the folder where endpoints are stored. Similar to all the other endpoints, you need to create a new folder for your endpoint (e.g. “some_alert_endpoint”): for telegram we created a folder named telegram_endpoint. Then you need to create two files named

• http_lint.lua
  This file is used to check (this is know as linting) the values of the parameters that will be passed to the endpoint through the ntopng web interface. This file needs to have a single function called script.getAdditionalParameters(http_lint) and has to return the checks needed to be done on the inputs.
  
• manifest.lua
  It contains the title, the description and the author of the plugin.

Inside the telegram endpoint you also need to create a templates folder that will contain both the endpoint and recipient html templates. These two templates are used to show the various input arguments, descriptions, ecc. on the GUI
respectively of the recipient and endpoint configuration.

The various template parameters (format, inputs and so on) shown on the GUI, are taken from the locales folder divided according to the respective language file such as en.lua (English), it.lua (Italian) and so on. ntopng will choose the language set in the preferences panel for the ntopng web user.

Furthermore, there is a new directory named alert_endpoint folder, where we’re going to store the endpoint code. In this folder create a file named telegram.lua, and put here all the code necessary to send telegram messages, format the parameters and so on.

The first part of this script is depicted below.

and it defines the HTTP parameters used in the recipient and endpoint forms used by telegram. Inside the XXX_params you need to specify the parameters passed in the input forms: the endpoint_params will be used by the endpoint form template, and the other for the recipient form. Additionally it is required to create a function called telegram.format_recipient_params(recipient_params) responsible to format the output of the recipient parameters.

At this point you can create a function to be called periodically responsible to process the queued alerts (generated by ntopng when specific traffic patterns are detected, e.g. when a host contacts a malware site), dequeue and deliver them to telegram. This function is named telegram.dequeueRecipientAlerts().

In the above code the ntop.recipient_dequeue() function is the one that will dequeue triggered alerts that need to be sent via telegram.

Once notifications have been dequeued, they can be delivered to Telegram via a HTTP POST according to the JSON format specified by the Telegram API.

We’re almost ready: we just need to create an additional function that is used by the ntopng engine when during recipient configuration, the administrator wants to check if the setup is correct by sending a test message.

That’s all. Your telegram endpoint is now completed. As you can see most of the work is required to create the web template and format the message that will be sent via HTTP POST. As most systems now feature a REST API, we believe that integrating additional endpoints should be pretty simple by cloning the telegram endpoint code, renaming telegram to my_endpoint and doing little cosmetic changes. Are you ready to contribute to the ntopng development  by creating a new endpoint?

Enjoy!

This year due to the pandemic, we had to cancel our scheduled community event. Considered that we have introduced many new features in our tools we would like to invite you to an online mini-conference divided in three distinct events. The first event is a general even where we briefly summarise what we have done in the individual tools so people can have an overview of what we have done and where we would like to go. The other two events are instead focusing on specific tools so people can join to learn more in details what are those features about, what problem they address, and how to use them. Below you can find the individual events schedule.

November 24th

[Calendar: miniconference, Webinar URL, Duration: 90 min, 4 PM CET/10 AM EST]

• Introduction: overview of 2020 changes and improvements, 2021 roadmap.
• Update on nProbe 9.2, nProbe Cento 1.12, and nDPI 3.4
• Overview of new features of ntopng 4.2
• How to produce and deliver alerts to recipients and consumer applications
• How to monitor ICS/Scada networks with ntopng
• New features introduced with n2n 2.8
• Update on PF_RING 7.8 and n2disk 3.6
• Public discussion with the ntop community

December 3rd

[Calendar: ntopng, Webinar URL, Duration: 90 min, 4 PM CET/10 AM EST]

• Using ntopng 4.2: real life scenarios that highlight how new features can be used in practical use cases
• How to implement ntopng endpoints and alert scripts, for extending ntopng
• Using ntopng Edge in production
• Embedding ntopng in Cubro EXA8

December 10th

[Calendar: nprobe_n2disk, Webinar URL, Duration: 90 min, 4 PM CET/10 AM EST]

• Traffic monitoring: how to use probes to deliver monitoring data to ntopng and external  applications.
• Embedding nProbe: low-end OpenWRT-based systems and 40-100 Gbit cPacket cProbe
• Mastering n2disk for efficient packet-to-disk, 100 Gbit and metadata generation

Notes

• All events will be online (English language)
• They are scheduled at 4 PM CET / 10 AM EST in order to enable everyone to join.
• All events will be recorded in case somebody misses them.
• You do not need to register to attend them: just click on the event webinar URL (each event has a different URL)

Precise packet timestamping is a key feature for network traffic analysis and troubleshooting. Traditionally many people use FPGA-based NICs with precise timestamping (e.g. Napatech, Silicom) even though a good precision can be obtained with PTP-based NICs such as many Intel network adapters. A better alternative to this practice is to avoid ad all using specialised adapters and rely on existing network devices to timestamp packets.

Arista packet brokers with MetaWatch  can be configured to add an extra trailer (Metamako) with metadata to every captured packet. In fact Arista 7150 Series devices are able to add packet trailers and generate keyframes to provide high-resolution timestamping, allowing for advanced network analysis and precise latency measurements. Arista MetaWatch devices are also able to include device information such as the Device ID and the incoming Port ID for captured packets and thus identify packet source that is then propagated to packet consumers. Below you can see an example of the packet trailer containing this information.

n2disk, ntop’s software tool for packet recording, is able to dump traffic and build an index on the fly, for enabling quick traffic retrieval by specifying the time interval and a BPF-like criteria. In addition to the 5-tuple, n2disk is able to index extended metadata, including those provided by MetaWatch devices and use the timestamp reported in the packet trailer. n2disk stores the device <ID> and interface <ID> in the packet index and it allows to use then when running traffic extractions.  A typical use case is the ability to retrieve traffic that went through a specific port and at a specific time in our network, this in addition to traditional IP address, port and layer-7 based filters.

This can be enabled in n2disk by adding –extended-index 4 and –hw-timestamp metawatch to the configuration (for further information please read the documentation). Example:

--interface=eth1
--dump-directory=/storage
--timeline-dir=/storage
--disk-limit=90%
--index
--extended-index=4
--hw-timestamp=metawatch
--index-on-compressor-threads
--reader-cpu-affinity=0
--compressor-cpu-affinity=1
--writer-cpu-affinity=2

n2disk is also able to export flow metadata to ntopng, acting as a flow probe, similar to what nProbe or nProbe Cento do. In this configuration, when support for MetaWatch devices is enabled with –hw-timestamp metawatch, n2disk also exports device and port information by populating %INPUT_SNMP %OUTPUT_SNMP %OBSERVATION_POINT_ID Information Elements. Example n2disk configuration file:

--interface=eth1
--dump-directory=/storage
--timeline-dir=/storage
--disk-limit=90%
--index
--extended-index=4
--hw-timestamp=metawatch
--index-on-compressor-threads
--reader-cpu-affinity=0
--compressor-cpu-affinity=1
--writer-cpu-affinity=2
--zmq=tcp://127.0.0.1:5556
--zmq-export-flows

ntopng configuration file example:

-i=tcp://*:5556c

In this configuration, device and ingress/egress port information are collected by ntopng and displayer in the flow details page as depicted below as well hardware timestamps are used in dumped pcaps.

In summary, thanks to Metamako support in ntop tools it is possible to combine precise timestamping as well packet-to-disk with real-time monitoring capabilities. This is an improvement with respect to hardware-based timestamping NICs that provide just timestamping and offer no device visibility and match on the actual network topology, very useful feature to locate and troubleshoot network issues.

Enjoy!

Earlier this month a new highly evasive malware attacker named SunBurst has been disclosed. Immediately some countermeasures have been disclosed and in particular some Snort/Suricata rules have been published. We have analysed the rules trying to figure out if ntop tools could detect and block Sunburst and the answer is yes, you can. Let’s have a look at some of the rules. The first thing you can observe is that the rules are any/any, meaning that an IDS has to look into every single connection this because most IDS do not use DPI an ntop tools do, hence they need to search everywhere instead of targeting the exact fields: this means that overall the tool performance is degraded as even traffic that is not relevant has to be analysed, and that you might encounter false positives.

The rules below

alert tcp any any <> any 443 (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"|16 03|"; depth:2; content:"avsvmcloud.com"; distance:0; sid:77600845; rev:1;) 
alert tcp any any <> any 443 (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"|16 03|"; depth:2; content:"|55 04 03|"; distance:0; content:"digitalcollege.org"; within:50; sid:77600846; rev:1;) 
alert tcp any any <> any 443 (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"|16 03|"; depth:2; content:"|55 04 03|"; distance:0; content:"freescanonline.com"; within:50; sid:77600847; rev:1;) 

are basically a TLS SNI (Server Name Indication) match.

that you can detect with nDPI.

Note that such rules are suboptimal as they have been designed before encrypted traffic was used, and so they are very primitives and limited in scope. See for instance what DPI reports for such TLS traffic

TCP 192.168.1.102:51293 <-> 20.140.0.1:443 [proto: 91/TLS][cat: Web/5][7 pkts/998 bytes <-> 6 pkts/1553 bytes][Goodput ratio: 52/74][1.74 sec][ALPN: h2;http/1.1][bytes ratio: -0.218 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/109 253/420 1142/1033 447/434][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 143/259 583/1215 180/428][Risk: ** Self-signed Certificate **][TLSv1.2][Client: avsvmcloud.com][JA3C: 2a26b1a62e40d25d4de3babc9d532f30][JA3S: 364ff14b04ef93c3b4cfa429d729c0d9][Issuer: CN=localhost][Subject: CN=localhost][Certificate SHA-1: D2:D1:B8:2B:15:FB:C9:51:B7:24:FF:56:B4:EF:9D:82:E2:E5:EA:B3][Validity: 2020-10-14 21:20:12 – 2022-12-17 11:32:25][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0,0,0]

As you can see this is a self-signed TLS certificate that is not a good thing either.

Other rules like these below are different but similar

alert tcp any any -> any any (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"T "; offset:2; depth:3; content:"Host:"; content:"freescanonline.com"; within:100; sid:77600852; rev:1;) 
alert tcp any any -> any any (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"T "; offset:2; depth:3; content:"Host:"; content:"deftsecurity.com"; within:100; sid:77600853; rev:1;) 
alert tcp any any -> any any (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"T "; offset:2; depth:3; content:"Host:"; content:"thedoccloud.com"; within:100; sid:77600854; rev:1;) 
alert tcp any any -> any any (msg:"APT.Backdoor.MSIL.SUNBURST"; content:"T "; offset:2; depth:3; content:"Host:"; content:"virtualdataserver.com"; within:100; sid:77600855; rev:1;)

In this case these rules basically say: search on HTTP (even on non standard ports) and in case you find connections towards specific sites (e.g. freescanonline.com) raise an alert.

In summary, these are old-style rules designed for year 2000 protocols that need to be refreshed. These are the equivalent rules for nDPI

$ cat sunburst.protos
#  Format:
#  <tcp|udp>:,<tcp|udp>:,.....@
#  Subprotocols
#  Format:
#  host:"",host:"",.....@
#
#  IP based Subprotocols
#  Format:
#  ip:,ip:,.....@

host:"avsvmcloud.com"@APT.Backdoor.MSIL.SUNBURST
host:"digitalcollege.org"@APT.Backdoor.MSIL.SUNBURST
host:"freescanonline.com"@APT.Backdoor.MSIL.SUNBURST
host:"freescanonline.com"@APT.Backdoor.MSIL.SUNBURST
host:"deftsecurity.com"@APT.Backdoor.MSIL.SUNBURST
host:"thedoccloud.com"@APT.Backdoor.MSIL.SUNBURST
host:"virtualdataserver.com"@APT.Backdoor.MSIL.SUNBURST

and you can now start ndpiReader as follows

$ ndpiReader -p sunburst.protos -i ~/avsvmcloud.com.pcap -v 2

...

Detected protocols:
APT.Backdoor.MSIL.SUNBURST packets: 13 bytes: 2551 flows: 1

Protocol statistics:
Acceptable 2551 bytes

JA3 Host Stats:
IP Address # JA3C
1 192.168.1.102 1

1 TCP 192.168.1.102:51293 <-> 20.140.0.1:443 [proto: 91.255/TLS.APT.Backdoor.MSIL.SUNBURST][cat: Web/5][7 pkts/998 bytes <-> 6 pkts/1553 bytes][Goodput ratio: 52/74][1.74 sec][ALPN: h2;http/1.1][bytes ratio: -0.218 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/109 253/420 1142/1033 447/434][Pkt Len c2s/s2c min/avg/max/stddev: 66/66 143/259 583/1215 180/428][Risk: ** Self-signed Certificate **][TLSv1.2][Client: avsvmcloud.com][JA3C: 2a26b1a62e40d25d4de3babc9d532f30][JA3S: 364ff14b04ef93c3b4cfa429d729c0d9][Issuer: CN=localhost][Subject: CN=localhost][Certificate SHA-1: D2:D1:B8:2B:15:FB:C9:51:B7:24:FF:56:B4:EF:9D:82:E2:E5:EA:B3][Validity: 2020-10-14 21:20:12 - 2022-12-17 11:32:25][Cipher: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384][Plen Bins: 33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,33,0,0,0,0,0,0,0,0,0,0,0,0]

Now you can use this technique in other tools such as ntopng as follows

ntopng -p sunburst.protos -i ~/avsvmcloud.com.pcap

Then inside ntopng you have to tell it that Sunburst is a malware by binding (menu Settings -> Applications and Categories) that Sunburst belongs to the malware category.

Done that ntopng detects it as a malware

and triggers an alert

that can be sent via the endpoint/recipients mechanism to external applications, messaging apps, ElasticSearch or SecurityOnion.

If in addition to detecting you want also to block it, just use ntopng Edge (that is basically ntopng inline) and you’re done.

Enjoy !

Creating Grafana dashboards out of ntopng data basically boils down to:

• Configuring ntopng to export timeseries data to InfluxDB
• Configuring the Grafana InfluxDB datasource to extract timeseries data from InfluxDB
• Adding Grafana Dashboards panels with ntopng data

This post aims at covering the topics above to serve as reference for those who want to create Grafana dashboards.

Configuring ntopng to Export Timeseries Data to InfluxDB

To configure ntopng to export timeseries data to InfluxDB, visit the ntopng Timeseries preferences page, and pick InfluxDB as driver. Then, it suffices to configure InfluxDB connection parameters. Once preferences are saved, ntopng will start exporting timeseries data to InfluxDB.

Configuring the Grafana InfluxDB Datasource

The same InfluxDB connection parameters specified above to configure ntopng can be used also to create a Grafana InfluxDB datasource. To create the datasource, pick the Datasources entry under the Grafana configuration menu, and add a new datasource of type InfluxDB. Then, it is enough to specify InfluxDB connection parameters.

Clicking on “Save & Test” will automatically test the connection and save it.

NOTE: The Grafana ntopng plugin datasource is outdated and should not be used.

Adding Grafana Dashboards panels with ntopng data

Now that Grafana is properly set up to extract timeseries data from InfluxDB, new panels with ntopng timeseries data can be added to dashboard panels.

Timeseries data are added to panels using the Grafana query builder. The query builder helps constructing the classical SELECT-FROM-WHERE clauses to pick the right data.

Different queries need to be constructed, depending on whether a gauge or a counter is being charted. Gauges and counters are the two types of timeseries exported by ntopng:

• Gauge are for things like the number of active flows, or active hosts
• Counters are for continuous incrementing values such as bytes sent and received

Gauges

Likely, to chart a gauge, a query will have to take the non_negative_derivative. Indeed, continuous incrementing values are only meaningful when derived, that is, when compared with their adjacent values. For the sake of example, a gauge panel with interface traffic can be created as follows.

Counters

To chart counters, there is no need to take the derivative. Data can be taken as-is. For example, one can create a panel with the number of active flows for an interface as follows.

A Complete Dashboard

A complete yet basic dashboard can look like the following.

The dashboard can be downloaded in JSON format from this link.

Enjoy !

This is to announce the immediate availability of both ntopng and nProbe for OPNsense, pfSense and FreeBSD, directly supported by ntop, with nightly builds and all the features present on all other supported platforms such as Linux, Windows and MacOS. You can now

• Monitor network traffic based on nDPI.
• Encrypted traffic analysis (ETA) that enables you to have visibility of encrypted traffic and answer to questions such as: what portion of my available bandwidth is used by Netflix?
• Cyber threats analysis: ntopng con be used to effectively detect attacks, anomalies and malware including SunBurst, Qakbot and Trickbot.
• Support enterprise protocols such as NetFlow, sFlow, and SNMP, that can be combined with packet-based visibility.

Installation

You can download packages from https://packages.ntop.org/FreeBSD/ and once installed they can be administered from the GUI as shown in the picture below.

Video Guides

User’s Guides

We have created specific guides for OPNsense:

• nProbe
• ntopng

that can complement the product user’s guides.

Questions ?

You can contact the ntop team and the rest of the community by using our community support.

Enjoy !

PS. Stay tuned as soon we’ll have some exciting news to share.

We are proud to announce that a couple of talks have been accepted at FOSDEM 2021, one of the most important FOSS conferences in the world that this yar will take place online due to the pandemic. In the Network monitoring, discovery and inventory devroom we will give two presentations titled “Using nDPI for  Monitoring and Security” and  “ntopng network monitoring and discovery“.

In addition, ntop has been given a virtual stand to present its opensource-related activities. The ntop team will always be available in a chatroom and a series of guests will give lightning talks during the two-days of FOSDEM.

This is the full schedule, follow hyperlinks for instructions on how to join!

No registration is necessary, just join the session clicking on the event link.

See you soon !

Due to pandemic, many people are now working in a delocalised world: some work from home, others from the office. To make things even more complicated, in the past remote workers used to connect to the company network via a VPN. While this option is still possible, many resources are now available from the cloud thus making VPNs obsolete in some environments, in particular for mobile workforce that connects to the Internet by means of a cellular network.

In the past months, some people contact us to ask how they could:

• Monitor the traffic in remote networks such as a small site or home office.
• Continuously check the connection speed for every worker: namely perform periodic checks such as the active tests offered by ntopng (ping, continuous ping and speedtest) so that the company can check if the remote worker/site can profitably work having sufficient and permanent bandwidth available.
• Be able to do all this while remote users might be behind a NAT (e.g. when at home) or a CGNAT (e.g. when using a mobile network connection).

The figure below shows you how you can achieve that.

As most of you now, in addition to popular tools such as nProbe and ntopng, we develop a decentralised peer-to-peer VPN named n2n that allows you to create your own network meshed topology simply installing a small software named edge (and part of the n2n toolkit) and leveraging on a supernode (still part of n2n) that allows remote edge’s to connect even if behind a network. While people can use their private supernode, ntop offers a public supernode available at supernode.ntop.org in case you do not want to setup a supernode. Note that even in this latter case your data will be completely protected due to the end-to-end encryption provided by n2n. Using n2n you can in essence create a network overlay which crosses firewall and that allows you to reach remote network edges without paying attention to network constraints such as NAT. You can read more here about n2n configuration. Note that n2n nodes can speak directly if the NAT allows it, meaning that you might be able to do A->B directly while B->supernode->A

Once n2n is setup, you have standard network connectivity (ping, http etc.) and you can now monitor your network without setting up any VPN server. Now let’s make a simple monitoring example as in the above picture.

• Start nProbe on each satellite node as follows:
  • nprobe -i eth0 –zmq tcp://192.168.X.1:1234 (where X is the network of the remote site)
• Start ntopng at the central monitoring console as follows:
  • ntopng -i tcp://192.168.2.1:1234 -i tcp://192.168.3.1:1234 -i tcp://192.168.4.1:1234 -i “view:tcp://192.168.2.1:1234,tcp://192.168.3.1:1234,tcp://192.168.4.1:1234”

and this is the final result

Enjoy !

Telemetry protocols such as sFlow/NetFlow, SNMP or packet-based traffic analysis are the source of data for network traffic monitoring. For a long time visibility was the main issue and people were attracted by new tools such as Grafana that allowed them to put on a screen a lot of data. A big misconception in this area is that the more data you see the less you understand as networks are constantly growing in both size and complexity and adding second or a third screen a PC isn’t the perfect way to scale up in known what is happening in a network. The score is a numerical indicator (present in all ntopng versions) that when non-null it indicates that some kind of issue is present: the higher is the score the worst is the problem associated, and thus the higher is the attention that a network operator should put on this resource.

The main source of score are the flows that have three score values:

• The flow score that indicates how bad is this flow. For instance a flow with severe retransmissions has a non null flow score.
• The flow source host score: a numerical value associated with the source host. For instance if in the above flow retransmissions are only destination->client the source host score for this flow will be zero, but if the source has also retransmissions the value will be positive.
• The flow destination host score: same as before but for the flow destination host.

As flows can have multiple issues, each issue found on a flow (for instance a TCP retransmissions or an outdated TLS version detected on a flow) contribute to the score. Hence the flow/source/destination score is the sum of all individual scores found on such flow. The flow score is computed by the flow user scripts (Settings -> User Scripts) when they are executed on a flows.

The host score is computed as the sum of all flow scores where such host is either client or server, complemented with additional scores eventually found on this host. A typical example is a network scan: every individual flow part of scan might have a zero score, but as this host is doing a scan, this results on a non-zero score. The host score is used to compute the score of other elements such as the Autonomous System, a Network, an Interface score that is the sum of individual scores in an iterative process as described above.

Using the flow score you can fo to Maps -> Hosts Map to see hosts are creating (or receiving) most trouble. In the above picture if you move the mouse over the purple or salmon coloured circles, you will see the name of the host creating most problems as the circle radius is proportional to the score associated. Inside each host (please make sure that Settings -> Preferences -> TimeSeries -> Host Timeseries is set to full) you can see a timeseries of the host score that you can use to see how healthy is a host and when troubles have been detected.

The next challenge we’re working on is to implement a method that allows you to detect score changes and alert you when this happens. This to drive network analysts towards the problem without having to constantly watch hundred of micro-windows.

Enjoy !

Most people use nProbe and ntopng to collect flows using an architecture similar to the one below

where nprobe and ntopng are started as follows:

• nprobe -3 <collector port> -i none -n none —zmq "tcp://*:1234" --zmq-encryption-key <pub key>
• ntopng -i tcp://nprobe_host:1234 --zmq-encryption-key <pub key>

In this case ntopng communicates with nProbe over an encrypted channel and flows are sent in a compact binary format for maximum performance. If you do not need nProbe to cache and aggregate flows, you can also add --collector-passthrough on the nProbe side to further increase flow collection.

This setup is good as long as you have a reasonable amount of flows to collect according to the system performance. In case you have too many flows to collect (e.g. you have many routers sending nProbe, or a big router sending many flows per second) this setup does not no scale up because it does not take advantage of multicore architectures. The solution is to

• Distribute the load across multiple (instead of one) nProbe instance.
• On ntopng collect flows over multiple ZMQ interfaces and eventually aggregate them with a view interface.

In essence you need to adopt an architecture as follows

where you start nProbe/ntopng as follows:

• nprobe -i none -n none -3 2055 —zmq tcp://127.0.0.1:1234

• nprobe -i none -n none -3 2056 —zmq tcp://127.0.0.1:1235

• nprobe -i none -n none -3 2057 —zmq tcp://127.0.0.1:1236

• ntopng -i tcp://127.0.0.1:1234 -i tcp://127.0.0.1:1235 -i tcp://127.0.0.1:1236 -i view:all

In this case each nProbe instance will process flows concurrently and the same will happen on ntopng. All you need to do is to try to share the ingress load across all the nProbe instances so that each instance receives a similar amount of flows and the workload is shared across them.

Instead, in case you have a single router exporting flows towards the same nProbe instance, the above trick cannot be used so we need to load balance differently. This can be achieved as follows

where you configure nProbe this way:

• nprobe -i none -n none -3 2055 --collector-passthrough —zmq tcp://127.0.0.1:1234 —zmq tcp://127.0.0.1:1235 —zmq tcp://127.0.0.1:1236

• ntopng -i tcp://127.0.0.1:1234 -i tcp://127.0.0.1:1235 -i tcp://127.0.0.1:1236 -i view:all

In this case nProbe will load balance collected flows across multiple egress ZMQ and ntopng will collect flows concurrently as every ZMQ interface runs on a separate thread.

Please note that:

• The --collector-passthrough is optional but it can reduce the load on nProbe, that is desirable if you have a big router exporting flows towards the same instance.
• The above solution can also be applied to packet processing (nprobe -i ethX) and not just to flow collection (in this case --collector-passthroughis not necessary). Namely the more you spread the load across interfaces, the more you exploit multicore systems and your system can scale up.

Enjoy !

In this post Martin shows how he has used ntopng to detect Qakbot trojan. Many thanks for this contribution.

Introduction

I am using ntopng for network monitoring quite some time now and I was curios to see, what ntopng would alert when detecting malware.

The website malware traffic analysis is a great source for malware captured in network traffic. I decided to take a Qakbot infection with spambot activity [1]. From the pcap file name we see to expect

• Qakbot, a active [2] banking trojan [3]
• Cobalt Strike, a commercial penetration testing tool, unfortunately pirated versions of Cobalt Strike have been leaked [4]
• Spambot activity, distributing QakBot

1. Further I used a tutorial how to examine Qakbot infections with Wireshark from Unit42 [5].

For the analysis I setup a virtual machine with Ubuntu Server 20.04 LTS and installed the current ntopng nightly build [6], using the community license.

Ntopng per default is built for continuous network monitoring, but directing the interface to a pcap, it can read and process it.

Analysis

To start ntopng sudo is needed, followed by the -i for interface or pcap file:

sudo ntopng -i 2020-12-07-Qakbot-with-Cobalt-Strike-and-spambot-activity.pcap

The web gui can be reached over

http://localhost:3000 or http://<server ip>:3000

Ntopng will prompt for username and password and you will have to select a new admin password. Once logged in, you will see the interface overview, showing that the selected interface is a pcap dumb and the name of the pcap file:

A quick look at the traffic summary icons reveal lots of work ahead, and many problems are being identified:

Now rather than checking out the flow warnings or errors, I navigate to the “Host Map” from the Maps menu and select “Alerted Flows” in the dropdown filter menu. This view gives a fast overview of hosts with alerts. The hosts map shows on the x-axis hosts identified as servers with alerts and in the y-axis host identified as clients. Note: a host can a be a server and/or a client. The size of the circles is proportional with the number of alerts, i.e. the biggest circle shows the host with the most alerts. In the filter dropdown you can find other useful filters, like “Traffic Ratio” or “DNS Queries vs Replies”.

Host overview

I proceed with the biggest circle, the top left one, as the host has around 170 alerts. A mouse over reveals the IP address of 10.12.7.101, clicking on the circle opens the host information page:

Regarding security concerns, I am interested in the above market items:

1. The host is a Windows 8 client
2. The host has a security score as a client of 19’905, for a single host a very high number. Ntop uses scores as a risk-based approach to security or protocol related flaws. E.g. malicious data in a flow adds 210 to the host score, suspicious data in a flow adds 200. The idea is not to have absolute values or limits, like above 1000 is bad, below 100 all good, instead, the score can be seen as an indicator of possible problems.
3. Out of 634 flows from or to this host, 168 are with an alert. That means that approx. 1/4 of the flows have an issue. Very unusual indeed.
4. For this host 152 SMTP (or SMTPS) flows were counted. This means the user is connected to many different SMTP servers. Again, this is very unusual for a client. A average user maybe has 5 or 10 different email accounts, leading to 10+ SMTP flows, but 152 SMTP connections do not look good at all.

Now let us check the alarms generated by ntopng. By clicking on the caution sign  , the flow alerts of the host view opens:

Host flow alerts

The list I filter by score, in order to get the alerts with highest score.

Note:

We could further filter the list by “Severity”. But with this host it is not needed, basically there are only Errors, with the highest severity.

Let us have a closer look at the first error, “TLS Certificate Self-signed”. We can drill down by pressing the Magnifying glass Button:

Flow Details

Here again there is a lot of information about the flow. For the security analysis we take a closer look at the three highlighted items:

1. The extracted url of the client request, hXXp://amajai-technologies[.]world/ looks at first sight accurate, maybe the TLD .world is for a technology provider a bit off. But doesn’t it look similar to what it actually tries to copy, akamaitechnologies.com? Look-alike domains [7] are easily overseen in daily work and are a real cyber security threat. 
2. The TLS certificate is self-signed: Red flag. No reputable or professional service uses a self-signed certificate for external / internet communication. Furthermore the TLS issuer and subject fields are empty, the red flag just got bigger. For reference, how the certificate issuer information looks like:
3. Assumptions that something is odd with this flow are validated by the JA3 [8] client fingerprint. The crossed circle, , indicates that the client fingerprint is listed as malicious. Follow the JA3 to sslbl.abuse.ch for further details and yes, the client certificate is malicious:
   
   Although the blacklisted JA3 is not Qakbot, we just found out that this host is a well infected with the Gozi banking trojan.

Conclusion

Even for analysing recorded network traffic, ntopng is great help for getting an overview of what happened on the network. But the real value of ntopng is, it does all the scoring and alerting in real time. Facing above malware, I might not know immediately what exact type of malware it is, but the generated alerts are strong indicators that something with that host is not ok and it can be isolated from the network for further analysis.

Got curious? Get started and try to find the spam activity or the Qakbot flow(s) yourself!

We invite other members of our community to contact us if they want to share their work on this blog.

References

1. https://www.malware-traffic-analysis.net/2020/12/07/2020-12-07-Qakbot-with-Cobalt-Strike-and-spambot-activity.pcap.zip Note: the zip is password protected. Use infected to extract the pcap.
2. https://urlhaus.abuse.ch/browse/tag/Qakbot
3. https://malpedia.caad.fkie.fraunhofer.de/details/win.qakbot
4. https://www.zdnet.com/article/cobalt-strike-and-metasploit-accounted-for-a-quarter-of-all-malware-c-c-servers-in-2020/
5. https://unit42.paloaltonetworks.com/tutorial-qakbot-infection/
6. https://packages.ntop.org/
7. https://info.phishlabs.com/blog/what-is-a-look-alike-domain
8. https://engineering.salesforce.com/tls-fingerprinting-with-ja3-and-ja3s-247362855967

In  recent ntopng versions, alerts have been significantly enriched with metadata useful to understand network and security issues. In this post, we focus on the “Attacker” and “Victim” metadata, used to enrich flow alerts and label hosts. Specifically, the client or the server of a flow is labelled as “Attacker” when it is, with high probability, the originator of one or more security issue. Similarly, when the client or the server is deemed to be under attack, it is labelled ad “Victim”. For non security-oriented use cases (e.g. severe packet drops), hosts affected/originating the issue are still highlighted with a high score value but they are not labelled as attacker/victim as these words are used only in security.

For example:

• In a DNS request for a suspicious DGA domain name, the client is considered to be an “Attacker” as it is the originator or such potentially malicious request.
• When Remote Code Execution (RCE), Cross Site Scripting (XSS) and SQL injection attempts are detected, the client is the “Attacker” and the server is the “Victim” as it is being probed/attacked by the client who is trying to exploit its vulnerabilities.
• When a binary/.exe application is downloaded over HTTP pretending it is an harmless file such as a PNG image (see Introducing nDPI Risk Analysis), the client is considered to be compromised and the server is considered to be malicious as it is sourcing suspicious files so both are labelled as “Attacker”.
• A DNS data exfiltration alert has both its client and server labelled with “Attacker” as carrying out a data exfiltration requires both the client and the server to be running exfiltration software such as iodine.

In the reminder of this post, we analyse Hancitor Infection (pcap courtesy of Malware-Traffic-Analysis) in order to demonstrate the effectiveness of ntopng with its new “Attacker” and “Victim” metadata in detecting this kind of security issues.

An Hancitor infection is basically a multi-stage chain of events that causes the targeted host to download malware files actually turning it in a malicious host in the hands of a threat actor. This post discusses the details of this kind of infections.

Let’s see what happens when we use ntopng to analyse the pcap with the Hancitor infection. You can start ntopng as

ntopng -i 2021-06-01-Hancitor-with-Cobalt-Stike-and-netping-tool.pcap -m "10.0.0.0/8"

First of all, it emerges pretty clearly from the header bar that there’s something suspicious going on, as there are thousands of flows with errors and warnings. Note that the number of alerted flows can vary according to your configuration (see in the left sidebar menu “Settings” -> “User Scripts”)

Clicking on the red “Errors” badge brings us to the flows page, filtered by flows having errors. By opening the “Status” dropdown, it becomes quite clear there is some suspicious activity as highlighted by the several suspicious DGA domain requests as well as by 2000+ of suspicious file transfers.

However, this page is not enough to understand if there is an attack going on and who is the source of the trouble. This becomes more evident when visiting the flow alerts page. Before browsing the alerts, we can set a filter to only see security-related alerts having an “Attacker”

With this filter, ntopng shows only alerts for which it has detected an attacker. Indeed, Suspicious DGA Domain alerts start popping out. The skull symbol helps in identifying the “Attacker” that, in this case, is the Hancitor-infected Windows client host.

If we keep browsing the alerts with an attacker, we will also see thousands of Suspicious File Transfer alerts. For this kind of alerts, two skulls are shown. Indeed, it is reasonable to mark as “Attacker” not only the compromised Windows host, but also the server that is distributing malicious files.

But why does file transfers for seemingly innocuous files ga.js are deemed to be suspicious? Well, because in practice, these are not Javascript files! Sometimes, they are just empty files, some other times they are binary files with unknown contents. This can be verified simply by extracting these ga.js files with Wireshark and demonstrates the effectiveness of ntopng in detecting these suspicious transfers taking place in the network. Additional suspicious files can be discovered simply by browsing these alerts. Those are requests to download Ficker Stealer and Cobal Strike.

You can identify attacker/victim in int Host Map (left sidebar menu “Maps” -> “Hosts”)

as they are the two outliers in the X/Y axis. Clicking on the bubble you can immediately jump to the host page and see what happens, as already discussed in this article.

Finally, it is worth mentioning that “Attacker” and “Victim” metadata are available also for host alerts, in order to spot hosts doing suspicious activities such as SYN scans, or high volumes of DNS or ICMP traffic. In addition to use the score in realtime as shown above, you can also monitor the score overtime in order to detect when it deviates from its expected behaviour. This is compulsory to detect “soft” behavioral changes that wouldn’t be detected with the score technique shown above, that instead is used to spot attackers with a more aggressive behaviour.

In summary, we have implemented in ntopng both “signature based” (i.e. by means of alerts) and “behavioural based” (i.e. metric analysis overtime to detect deviations from expected behaviour) anomaly detection. In a future port, we’ll elaborate more in detail how behavioural changes are detected and reported.

Enjoy !

Introduction

ntopng, in combination with nProbe, can be used to collect NetFlow. Their use for NetFlow collection is described in detail here.

In this post we measure the performance of nProbe and ntopng when used together to collect, analyze, and dump NetFlow data. The idea is to provide performance figures useful to understand the maximum rate at which NetFlow can be processed without loss of data.

Before giving the actual figures, it is worth discussing briefly the most relevant unit of measure that will be used, i.e., the number of flows per second – fps in short. NetFlow is per se a protocol designed to export communication flows. It carries flows inside UDP datagrams, and each UDP datagram carries multiple flows. For this reason, it is natural to choose the number of flows per unit of time when it comes to analyzing the performance of ntopng and nProbe when dealing with NetFlow. Indeed, we will use fps to quantify the number of flows analysed and written to disk when giving performance figures.

Knowing how much your NetFlow actually accounts for, in terms of fps, is also fundamental to make sense of the figures we are giving and to understand if ntopng and nProbe perform well enough for your network. In general, a few Mbps of NetFlow are able to carry thousands of flows per second. If you do not know the how many flows per second your NetFlow accounts for, you can consider that, as a ballpark figure, 10 Mbps of NetFlow v9 carry approximately 12,000 fps. Note that this is just a ballpark as the flow size depends on the flow template for v9 and IPFIX.

Performance Figures

Finally, it’s time to look at the figures, that are given both for two and four interfaces.

The rationale of expressing performance numbers for a variable number of interfaces is that multiple interfaces:

• Can be used to effectively balance the NetFlow across multiple cores
• Can be used to keep the traffic of multiple NetFlow sources separated

However, to give consistent numbers, results are also expressed in terms of fps per interface in the first column. Note also that results apply when using interface views too and that, when nIndex flow dump is enabled, the expected reduction in the processing rate is between 2 and 3 Kfps.

Example

To give a real example that can help you in making sense of the figures above, let’s consider a case where you want to collect NetFlow at 85 Mbps, i.e., approximately, 100 Kfps. To collect at 100 Kfps, according to the table above, we need at least 3 interfaces. Let’s set up an ntopng instance with four interfaces to say on the safe side – traffic is aggregated together using the view with -i view:all.

./ntopng -i tcp://*:5556c -i tcp://*:5557c -i tcp://*:5558c -i tcp://*:5559c -iview:all -F"nindex"

Assuming NetFlow is arriving on port 2056 and ntopng is running on host 192.168.2.225, we can configure nProbe to collect NetFlow and load-balance flows on the four interfaces above as follow

 ./nprobe -i none -n none --collector-port 2056 -b 1 -T "@NTOPNG@" --zmq tcp://192.168.2.225:5556 --zmq tcp://192.168.2.225:5557 --zmq tcp://192.168.2.225:5558 --zmq tcp://192.168.2.225:5559 --zmq-probe-mode --collector-passthrough

Final Remarks

In this post we have seen the performance figures of ntopng and nProbe when used to collect NetFlow. We have seen how to quantify the flows per second (fps) carried by NetFlow and have also determined that the combination of nProbe and ntopng is suitable for the collection of NetFlow at 100+ Kfps. Figures given are valid for the latest ntopng 4.3 (soon ntopng 5.0 stable) and nProbe 9.5 (soon 9.6 stable), that represent a significant step towards high-speed flow collection. Indeed, their performance exceeds the performance of the previous versions by at least a 15%.

You can read more about high-speed flow collection here or directly refer to the user guide.

In our previous post we have analysed the performance of the pipeline nProbe+ntopng for those who need to collect flows and analyse them, trigger alerts, create timeseries, provide a realtime monitoring console, dump them to nIndex and inform remote recipients in case of some problem is detected. This is the main difference between the ntop solution and a NetFlow collector whose main goal is to dump flows on a database with any or little flow analysis. In essence the current state of the art with 4 nProbe instances sending data to a single ntopng instance is a process rate of ~100k flows/second.

In this post we analyse the performance of nProbe alone, so that you can make up your mind if:

• You only need to collect flows and dump them on disk or on a database.
• You want to create a distributed collection architecture where you have multiple collection points and you need to know how fast you can scale.

Goal of this article is to show you the various options we can offer you in terms of flow collections and processing, and show you that maximum performance you can achieve on a single stand-alone system (no clustering or multi-node deployment) when collecting hundred of thousand flows/sec and dumping them on a open-source database able to store trillion of records and retrieve them with sub second queries.

nProbe: How to Dump Flows Into ClickHouse

One of the new features of the recent nProbe dev builds is the ability to dump flows in ClickHouse, a column-oriented database able to scale to hundred of billion or records/node using a single node and no more than 16 GB or RAM regardless of the number of records. The nice thing about ClickHouse is that you can both access this database using the native client (as nProbe does for importing data) or as a MySQL database listening on port 9004 (MySQL default port is 3306). As ClickHouse is integrated with Grafana, you can easily create dashboards with a few clicks if you want. Currently ClickHouse support is integrated in the nProbe Enterprise M and up versions (not available on Windows), and it will be integrated in the next stable nProbe release due by the end of this month. Similar to MySQL, you can enable ClickHouse in nProbe with:

 --clickhouse=<host[@port]>:<dbname>:<prefix>:<user>:<pw>

Example if you have a the same box nProbe and ClickHouse running you can start nProbe as

nprobe --clickhouse=127.0.0.1:ntop::default:helloworld

where ntop is the database name, default is the ClickHouse username with password helloworld. nProbe stores temporary data (used to batch data for ClickHouse import) in the directory specified with -P or on a temporary directory if -P is not specified.

As already said, once your data is on ClickHouse you can query it using mysql client or the native client as follows:

clickhouse-client --password xxxxxxx 
ClickHouse client version 21.7.3.14 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 21.6.5 revision 54448.

ClickHouse server version is older than ClickHouse client. It may indicate that the server is out of date and can be upgraded.

pc-ntop :) use ntop

USE ntop

Query id: 8c8d9ea5-5396-41ee-a648-3524c3c22add

Ok.

0 rows in set. Elapsed: 0.001 sec. 

pc-ntop :) select count(*) from flows where IPV4_SRC_ADDR = IPv4StringToNum('192.168.0.1')

SELECT count(*)
FROM flows
WHERE IPV4_SRC_ADDR = IPv4StringToNum('192.168.0.1')

Query id: ce8c8375-aa35-4f94-a1ca-9a09d891902d

┌───count()─┐
│ 110894877 │
└───────────┘

1 rows in set. Elapsed: 0.035 sec. Processed 110.90 million rows, 443.58 MB (3.19 billion rows/s., 12.76 GB/s.)

pc-ntop :) Bye.

In a future ntopng version we will eventually add support for querying data directly from ntopng.

nProbe Performance

Below you can find a table with the nProbe performance we have measured on a Ubuntu 18.04 system based on Intel E3-1241 and a SSD drive where data is stored. In order to measure that maximum nProbe performance we have added a new testing CLI option named  –simulate-collection that simulates infinite flow ingress collection so that we can measure the worst case with randomly generated flows. The base command line option we have used (it needs to be slightly adapted according to the test case) is 

nprobe -b 1 –simulate-collection -t 10 -n none –clickhouse=127.0.0.1@9004:ntop::default:XXXXX -T “%IPV4_SRC_ADDR %IPV4_DST_ADDR %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %SRC_TOS %SRC_AS %DST_AS %EXPORTER_IPV4_ADDRESS %IPV6_SRC_ADDR” -P /tmp/ramdisk/

As you can see flows are dumped to /tmp/ramdisk/ (created with sudo mount -t tmpfs -o size=5G myramdisk /tmp/ramdisk) before are batch imported into ClickHouse that is running on the same host where nProbe runs. 

Below you can find the results of all test cases:

The above figures are per nProbe instance, so you can run multiple nProbe instances for maximising performance. In our tests with a local ClickHouse database, we can import up to 150k Fps with two nProbe instances that seems to be the maximum a single ClickHouse database can handle.

As a final note you can see, the figures above exceed the nProbe+ntopng performance of 100k Fps, but it is hard to compare them 1:1 as:

• nProbe+ClickHouse is just collecting flows and storing them on a high-performance MySQL-compatible database, as most collectors on the market do. No data analysis, alerting or traffic timeseries are created.
• nProbe+ntopng is a complete NetFlow traffic solution, with data analysis, anomaly detection, traffic behaviour analysis, timeseries etc.

So depending on what are your requirements you can choose one or the other. Note that you can use nProbe+ClickHouse+ntopng for the best of both worlds. We believe it is a great achievement to be able to collect, analyse and dump over 100k flows/sec on a single host with no need to create a cluster of database instances (necessary for instance if you want to dump flows with nProbe/ntopng into Elastic) or costly systems to both build and maintain.

Enjoy !