GitHub on Wednesday withstood the largest-ever recorded distributed denial of service attack in history, experiencing roughly 10 minutes of disruption during the onslaught, which was amplified using exposed memcached servers -- a vector that has seen a significant increase in abuse since last month.
A Mar. 1 blog post by the GitHub Engineering team reported that the attack reached a peak of 1.35 terabytes per second, with a throughput of 126.9 million packets per second. The all-out assault rendered the web-based hosting service unavailable on Feb. 28 from 17:21 to 17:26 UTC and intermittently unavailable from 17:26 to 17:30 UTC.
Nevertheless, the site staved off the volumetric attack and quickly restored normal service, an achievement GitHub credits to doubling its transit capacity over the past year, as well as the decision to move traffic to Akamai Technologies' content delivery platform, which was able to provide additional edge network capacity.
The post's author, GitHub's manager of site reliability engineering Sam Kottler, wrote that the attack "originated from over a thousand different autonomous systems (ASNs) across tens of thousands of unique endpoints."
The use of memcached servers continues a rapidly emerging trend that was initially referenced by content delivery network provider Cloudflare, which reported earlier this week that misconfigured, publicly accessible memcached servers were being exploited via UDP (User Datagram Protocol) port 11211 to deliver malicious traffic. (Memcached is a free and open-source, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.)
Cybersecurity software company Imperva on Mar. 1 issued a similar warning, reporting that its researchers observed two massive DDoS amplification attacks on Feb. 28 -- the same day as the GitHub incident. According to Imperva, these incidents also stemmed from memcached servers, which adversaries can recruit into DDoS attacks by sending a UDP packet "with a spoofed IP containing a message asking for statistics, which will cause the server to return an enormous message to the victim."
Ofer Gayer, senior product manager at Imperva, told SC Media via email interview that the attacks witnessed by his company targeted a cryptocurrency exchange, as well as e-commerce websites and enterprise networks.
Using this technique allows attackers to unleash a massive amount of packets for each small UDP packet they send, essentially amplifying the attack without having to rely on a botnet of hacked devices. While Imperva reported that the amplification factor can be 9,000x or more, both GitHub and Cloudflare wrote that it can actually soar as high as roughly 51,000x.
"The real issue here is not the DDoS attack itself, but the amplification mechanism used," said Mounir Hahad, head of threat research at Juniper Networks, in comments emailed to SC Media. "Memcached is a service that helps websites scale, but it is supposed to be internal and never exposed to the internet. Yet, there are a little over 90,000 misconfigured installations accepting non-authenticated requests coming from the internet that could easily use spoofed return IP addresses. That's a recipe for disaster."
"It behooves us to remediate this issue immediately before the next wave of attacks hits," Hahad continued. "In the meantime, it is recommended that all firewalls implement a rate limit on port 11211, or block it off completely."
On the other hand, Gayer said other recent DDoS attacks from the Mirai and Leet botnets are actually far more complex "since they generate legitimate-looking traffic at very high packet rates arriving from an endless pool of compromised IoT devices. The memcached vector uses amplification of a rarely used protocol, which makes it very trivial to mitigate..."
Moreover, the memcached servers used for amplification "are easy to detect and many cloud and transit providers are doing great work in actively filtering for outbound [memcached] traffic from their network and contacting server owners," Gayer continued. "Memcached servers are also typically operated by highly technical IT/webdev staff who can be contacted in order to update the version and remove the vulnerability, unlike IoT devices which are more of an uncontainable chaotic mess of end-user devices."
