Recent Downtime

The latest happenings on Smack Jeeves
Forum rules
Please stay on-topic when responding to news posts.

Recent Downtime

Postby Admin » January 9th, 2017, 3:15 pm

Hey Everyone,

In the last five days there have been a few periods of downtime ranging from a couple hours to 5 hours. I want to sincerely apologize for this - our goal is to have 100% uptime, and while our record is very good, we consider any amount of downtime to be unacceptable. I want to address this now and share with you what happened and what we're doing about it.

The short version:
The periods of downtime have been caused by a number of factors, all essentially stemming from what appears to be a low-level attack on our service. We have taken steps to address these issues including software and security upgrades and are continuing to monitor, assess, and address the situation.

The full version:
On January 5th there were a few hours of downtime due to what essentially was a database crash. Specifically the database was hanging on a query for no apparent reason - kind of like the database equivalent of the "blue screen of death." No data was affected. We were back after a few hours of downtime.

It was at this time that I noticed a high number of suspicious requests coming to the server, by some kind of poor-man's botnet or something similar. It's possible that this attack is what triggered the database lockup.

A few days later, on January 8th (yesterday as of this writing), the same thing happened again. This confirmed that this was not a one-time fluke - the likelihood is that the crash was caused by a combination of the attack and a bug in our database server, so after recovering from the crash, we upgraded our database software. Also at this time, our server management company ran a script to clear some problematic connections to our server that were likely coming from the attack source. An unfortunate side-effect of this was that this script also released the IP addresses from our server - and without our IP addresses, the site became entirely unreachable. It took several more hours to figure out what had happened and how to fix it. Now that we've been down that rabbit hole, that problem should never happen again.

We are still under attack by this mysterious source. Things are running smoothly despite the attack, and we have and are continuing to enhance our security to mitigate this and other potential DDoS-style attacks. Why anyone would want to attack a webcomic hosting service is beyond me, but we're dealing with it and will do whatever is necessary to shut it down.

Should we encounter any more issues with this, you can check our Twitter @smackjeeves for updates. Thank you.
User avatar
Admin
Site Admin
 
Posts: 1259
Joined: August 17th, 2005, 11:10 pm

Return to News

Who is online

Users browsing this forum: Yahoo [Bot] and 2 guests