As more people are realising that in today’s cyber climate Distributed Denial of Service (DDoS) attacks are a matter of when, not if, the most common question I get asked is “What can I do to prepare?” I like to break it down into five key steps enterprises can take now to be prepared for a future attack.
- Centralise Data Gathering & Understand Trends
This is true across all security topics, but the last thing you want to be is blind when a DDoS attack hits. Generally the DDoS attack timeline goes something like this for the head of network operations:
9:00 am – your monitoring system starts lighting up like a Christmas tree and your phone is blowing up with SMS alerts saying “the site is down.”
9:01 am – your CEO calls you screaming “why is the site down?!?!?!?!”
Hopefully, you can answer that question, but without proper metrics and data gathering you can’t possibly hope to identify the root cause. It could be a network circuit down, data centre failure, DDoS attack, etc. With proper data gathering and monitoring in place, you can quickly identify a DDoS attack as the cause, and you can start the process of getting the website back up and running. It’s critical to identify the cause early as DDoS attacks can be quite complex and the sooner you jump on identification and remediation, the sooner the site will be back up.
At minimum, the metrics you should gather include:
- Inbound and outbound bandwidth on all of your network circuits, peering connections, etc
- Server metrics: CPU load, network and disk I/O, memory, etc
- Top talkers: top sources and destinations of traffic by IP and port
- If you are running a web site, you need to understand items like top URLs being requested (vs. the top URLs usually being requested), top HTTP headers, HTTP vs. HTTPS traffic ratios etc.
All of these metrics (and there are many more I didn’t cover) should then be sent to a central logging and correlation system so you can view and compare them from a single viewpoint. This helps you spot trends and quickly identify the sources and method of the attack.
This is especially important when it’s a very complex attack where it might not be an obvious issue (e.g. it’s easy to see when your network bandwidth is saturated, but when it’s a botnet simulating clicking the “Add to Cart” button to overwhelm your database resources, that isn’t as easy to spot; especially if you are trying to piece data from many disparate systems).
- Define A Clear Escalation Path
Now that you have determined it really is a DDoS attack, what next? Do you know who to call to get your service back up and running? What tools do you have in place to block the malicious traffic? If you have purchased DDoS protection (very smart!), how do you get the system fired up?
These are key questions that should be written down and answered BEFORE the attack hits. During an attack people are rarely calm and it’s no fun trying to figure out an escalation path in the middle of the craziness. Do it before the attack hits so you can calmly execute your plan and get your site back up and running.
Note that this doesn’t just mean “technical” contacts. You want to let the head of support and customer service know as well. You can bet customers will be calling in and there is nothing worse than to answer “weird, I didn’t know our site was down” when a customer calls. You also want to let your CEO know (if he or she doesn’t already).
Each business is different, so you should consider your situation and think of all the people who might want to know the website is down and add them to the list. An “outages” mailing list is a central place to report these items without you needing to remember who to send the info to every time.
If you do have a cloud-based DDoS protection service in place, make sure the group you have chosen internally to be the touch point with the provider has the up to date 24/7 hotline, email address to send capture files to, etc. The vendor should be one of the first calls you make to start the mitigation. You need to engage your mitigation provider immediately as they have done this many times before and will know what to do to get your site back up and running.
- Use Layered Filtering
In the discussion on size vs. complexity of an attack, you need to be able to handle both the “big and dumb” types (a whole lot of requests that are generally easy to spot as malicious – often known as “network level”) and “small and complex” (fewer requests, but extremely difficult to differentiate legitimate vs. malicious – commonly referred as “application level” or “layer 7″ attacks).
Some tools and techniques work (and scale) very well to mitigate against the “big and dumb” types, but fail miserably on the application attacks. On the other hand, some techniques that are required for application attacks have trouble scaling on the larger network attacks.
Recently, we have seen more of a third type of attack, “big and complex!” A combination of the two aforementioned attack types, these are big attacks where the traffic is really hard to identify as malicious or legitimate. With great technology and layered filtering though, you are in a better position to handle any of these types of attacks.
- Address Application & Configuration Issues
Not only are DDoS attacks really good at pinpointing bottlenecks in your network and security infrastructure, they are also amazing at identifying problems in your application; especially when it comes to performance tuning and configuration. If you haven’t done proper application load testing (both before launch and every so often to check for any slowness that may have crept in) a DDoS attack may be the first time your website or application has really been stress-tested.
You may find your database configuration is sub-optimal, or your Web server isn’t configured for enough open connections. Whatever the issue, you will quickly see how well you have tuned your website. It’s always a good idea to do load testing of your site on your schedule, not the attackers’.
- Protect Your Domain Name System (DNS)
This is crucial and yet probably the most overlooked of all of the above recommendations. I can’t tell you how many enterprises have spent millions of dollars on their Web hosting infrastructure (data centres, web servers, load balancers, database servers, etc.) but have only two low end DNS servers to handle all of their DNS traffic.
DNS is an extremely common target for DDoS attacks due to how critical the service is for Web availability (there are plenty of articles and examples of large Web properties going down due to DNS issues – often attack-related). If a customer can’t resolve the IP address of your website (which is the job of DNS), it doesn’t matter how much you have spent on your hosting, that customer is not getting to your site. Protecting your DNS as part of a good DDoS mitigation strategy is fundamental.
As Vice President of Strategy and Technology for the Verisign Network Intelligence and Availability (NIA) Group, Sean Leach is responsible for product and technical architecture, and strategy for the Group’s product line consisting of DNS hosting, threat intelligence, and DDoS protection services. Before joining Verisign, Sean was CTO of Name.com, a "top 20" domain registration and hosting company, where he was responsible for product and technical strategy, as well as engineering and operations management. Previously, Sean was Senior Director of Technology for NeuStar's Internet and Infrastructure Services Group (IISG), including the UltraDNS, Webmetrics, Registry, and all Internet and infrastructure related product lines. Sean has also served as Director of Product Engineering for UltraDNS (acquired by NeuStar in April, 2006), Director of Engineering for NetLojix/Impulse Internet Services, and Director of Engineering for one of the pioneering e-commerce application service providers. He is a sought after subject matter expert in DDoS, DNS, and cyber security, and has participated in the last two DHS sponsored Cyberstorm (cyber warfare simulation) events. Sean holds a BS in Computer Science from the University of Delaware and is currently pursuing research focused on DNS, Internet infrastructure, and combating the massive online crime epidemic.