amazon web services - AWS DNS DDoS mitigation

I have an AWS server that is currently under DDoS via DNS amplification. I've setup CloudWatch logs for the VPC ACL and it's logging an enormous amount of rejected DNS traffic. Despite that traffic being rejected, my primary server is unreachable.

I have a secondary server on the same VPC and subnet that can be reached without any problem.

Why is it that I can access one but not the other? The ACL should be filtering the traffic at the subnet level. So if one is unreachable then they both should be unreachable, but that's not the case.

And how does one mitigate a DNS amplification attack on AWS? AWS certainly has big enough pipes. Why is the ACL not doing the job?

Answer

I ended up solving the issue.

There were a couple issues actually. I had only blocked UDP port 53 (DNS) and as it turns out there were other ports being attacked. Since my server is just a web server I was able to block all UDP traffic in the ACL. That solved one side of the attack.

They were also overloading my web server with large post requests from compromised WordPress installs. I was able to add a few lines to my Nginx configuration that dropped requests from WordPress user agents and also block large post requests.

These were the settings I used in the http section of the Nginx config

    client_max_body_size 10k;
    client_body_buffer_size 10k;
    client_header_buffer_size 1k;

    large_client_header_buffers 2 1k;
    client_body_timeout 6;
    client_header_timeout 6;
    keepalive_timeout 5;
    send_timeout 10;

Then in the server section of the Nginx config I setup a drop for WordPress and wget initiated requests

if ($http_user_agent ~* (wordpress|wget)) {

   return 403;
}

These settings made the server a lot more difficult to bring down

I also used iptables to rate limit incoming http connections

# Rate limit new connections to port 80
-A INPUT -p tcp -m recent --dport 80 -m state --state NEW --set

-A INPUT -p tcp -m recent --dport 80 -m state --state NEW --update --seconds 20 --hitcount 20 -j DROP

Then I used iptables to limit the maximum number of simultaneous connections to port 80.

# Limit concurrent connections for a class B to port 80
-A INPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -m connlimit --connlimit-above 10 --connlimit-mask 16 --connlimit-saddr -j REJECT --reject-with tcp-reset
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

These things together made my servers much more difficult to DDoS. I'm now using multiple front end servers to reverse proxy requests to a backend server. I setup DNS round robin to expose the multiple IP addresses. This last additional step increased the amount of total bandwidth I could handle in an attack that got past all the other defenses.

So far the remaining attacks have not been able to take down my server.