We had an interesting outage today on one of our client's
websites. Out of nowhere, the website was inaccessible. The website runs by itself on a
dedicated physical Windows 2003 R2 server (probably overkill, I know, but that's a
discussion for a different day). After restarting IIS and ColdFusion Application
Service, the problem came back several times. My initial thought was that it was a DNS
issue, which happens occasionally - the last time it happened was after Hurricane Sandy
when we our ISP was out, and we had to make some network config changes. But, it was not
a DNS issue. My second thought was that it was a DDOS attack, but, there's very little
reason anyone would want to take this site down. When we called our ISP, the operator on
the other end noted that traffic was spiking significantly. As it turned out, the client
had unintentionally caused a DDOS on the website, after they FTPed a very large video
file, and then mass emailed a link to it. Hundreds of people clicked the link and
brought the site to its knees.
I am primarily a
Website Programmer, but I often have to contribute to server administration at times.
Sadly, I'm the resident ColdFusion and IIS expert, but I don't have a lot of experience
with this issue. What are some basic steps that I can take to prevent this from
happening in the future, since we cannot always control what files the client posts to
the website.
Here are some ideas I had, but I'm
unsure of the
impact:
- Limit the number
of connections in IIS. - Put media files on a
separate server (like an Amazon site, etc.). - File
requests of this type currently behind a server-script (i.e.
/www.site.com/viewFile.cfm?fileId=1424545, where the fileId references a file off the
webroot) that logs requests, and pushes the file to the browser using CFCONTENT. I could
edit this script to reject requests when they exceed a certain amount in a given
time-frame (i.e. a 5MB can be accessed globally 10 times in an hour). This may cause
some users frustration, but, if hundreds of users are attempting to view the file, the
site is going to crash anyways, as it did today, which is way more frustrating, since
there is no "pretty" message explaining why they can't get to the
file. - Update Request Tuning settings in ColdFusion
Administrator.Maximum number of simultaneous Template requests
is currently set to 20. I could reduce this number to something like 5 just to prevent
occurrences like this, but that would likely have an adverse affect on normal use of the
website.
I'm open to any
suggestions, as I'm continuing my research to report to the CTO with the best options,
so that we can put a solution into effect.
Thank
you.
UPDATE: Usage Report from the time
surrounding the outage:
src="https://i.stack.imgur.com/tEQot.gif" alt="Usage Report">
Comments
Post a Comment