The internet has come under a sustained and significant threat from network malware, especially since the emergence of the global Windows network worm in 2001 with Code Red and Nimda. Prior to this, most network worms were on a small, localized scale.
Since then, network malware has become a global web threat. For instance, in September 2001, the Nimda worm disrupted global BGP routing tables for hours at major internet peering points, and in January 2002 the SQL Slammer worm caused significant outages and slowdowns. DDoS attacks directed at the root DNS servers, most recently in February 2007, were launched only with the help of internet-scale malware and botnets.
Discovering this malware and other malicious activity is key to any global monitoring approach, especially an early warning system. Honeypots are often an excellent source of data, but they rely on the attacker encountering the system. By the same token, absorbing all of the data by active client collection techniques simply doesn't scale. Clearly a balance must be found, one that can be used to highlight possible new sources of activity.
Darknets, a global approach
A new approach to monitoring malicious internet traffic on a global basis is to utilize the darkest of ISP's. Darknets are unallocated IP space owned by the parent service providers or enterprises. It is the portion of the internet that has not been assigned to customers, and every network has some portion of darknet space. The advantage of putting sensors in darknets is that because it is unallocated, there are only two reasons for traffic to be going there, either some form of misconfiguration or bots and other malware. Darknets therefore have little background traffic, meaning that the data captured is purely signal and can be analyzed. The nickname comes from the fact that this portion of the network is not normally "lit up" with traffic hence it is usually dark. A key facet of any darknet is that it is globally routed and reachable, so a host anywhere on the internet sending traffic to it will register from any source.
Darknets work because network-scanning malware is unable to predict which addresses are in use on every level of the internet. Bots and malware are not intelligent enough to pick and choose where they go, they will simply attempt to spread to as many hosts as possible. Monitoring darknet traffic yields great visibility into what threats are present. Darknets allow ISPs to get the purest form of data from the heart of the internet.
At the global level, routing registries do not allocate every network octet, meaning significant chunks of the internet are not in use. Within an allocated network, not every subnet is in use, such as within an enterprise. And finally, within a subnet, not every address is used. By some estimates, only about one-third of the internet is in active use at any time, counting from the subnet that is DHCP allocated up through the BGP allocations given by organizations such as ARIN. This leaves tremendous room for darknets to be deployed.
Over the years, we have seen only a handful of internet worms try and avoid the largest of darknet monitors by carrying a list of networks to scan. This didn't work as well as the authors had hoped, and since then very few other malware authors have tried this. The bulk of bots and malware these days use "island hopping" strategies to locally bias their scanning and attacks, either by hardcoding such an algorithm (first made popular with Code Red II and Nimda) or through botnet scan commands focusing on the local networks. Even in these cases darknets observe the malware due to the sparseness of IP address assignments on the local network.
Multiple levels of data can be analyzed in darknets, including NetFlow-based approaches through honeypots. At the NetFlow level, routers and switches show what traffic is destined to a darknet by generating traffic summaries called "flows." This provides a lightweight data representation of the traffic by omitting payloads and aggregating packets into a single flow record, and typically consumes about one percent of the original traffic. These records are great for trend analysis and useful in analyzing global scan patterns. Packets can be collected, but if no listener is configured they typically provide no more data than a honeypot would. However, if a honeypot system is used, it can be used to discover the nature of the attack and provide further characterization.
Darknets are the network equivalent to email spam traps, dummy IM accounts to discover new attacks, and other such data collection points. The major differentiation between a darknet and a typical honeynet, however, is the scale of data collection. Darknets are not composed of one of two hosts but instead hundreds of addresses. This means that data analysis techniques have to scale up dramatically, focusing on trends and patterns instead of deep specifics.
At Arbor Networks, we have found that a distributed darknet monitoring system provides global visibility into malicious traffic and probes. The scan and attack patterns indicate the prevalence of bots and malware and a network of sensors continuously collects new malware samples. Because of this, there is usually an indication of a large-scale attack before it impacts customers too dramatically.
- Dr. Jose Nazario is a senior software and security engineer with Arbor Networks Security Engineering & Response Team (ASERT).
