+- Discovery Gaming Community (https://discoverygc.com/forums)
+-- Forum: Discovery General (https://discoverygc.com/forums/forumdisplay.php?fid=3)
+--- Forum: News and Announcements (https://discoverygc.com/forums/forumdisplay.php?fid=13)
+--- Thread: Downtime (/showthread.php?tid=146381)
Downtime - Alley - 12-19-2016
As you likely have noticed, Discovery was unavailable for the weekend.
When the problem initially appeared, I thought it was a mere crash or something similar and it would only take a few minutes for things to get back online. It's only shortly after that I realized the mess that just happened and that, well, there wasn't really any staff member capable of fixing something of this magnitude aside of me, so I'd have to sort this out.
TL;DR, an unwanted person was capable of logging on the web server using staff credentials and wiped everything. We honestly have no idea how these credentials were obtained and they've been instantly invalidated. For obvious reasons and to prevent witch-hunting, no name will be given. It can happen to anyone, even if you try to secure your stuff as best as possible. It's the dangers of the digital era and there are people that spend their time making software specifically to steal things, just like there are people making software to try to prevent it from happening. That's the era we live in.
The attack was done purposely to destroy the service. I was concerned there was an attempt at pulling information but nothing has been downloaded out of the box, which is some serious good news.
I doubt we will ever know who was really behind it as the attack routed through China, but the timing is certainly oddly specific as we were going to get rid of a specific someone. We'll likely never be given the real reason of this, it's possible this is not even related to disco and it was just someone that likes to see things burn.
Either way, things are back online. Somewhat thankfully, the attack was carried out after the automatic snapshot system I had put in place took a snapshot of the data disk at 17:00, the attack being carried out around 17:05.
The real reason for the extended downtime and the big discussion in the repurposed Corsair chat is that I hadn't been able to get the snapshot to restore for 8 hours and considering the other backup options were not put in place due to "reasons", we were essentially finding ourselves dumbfounded despite we had specifically attempted to prevent this kind of scenario from happening again. It essentially took me the entire weekend and this morning to finally figure out what was preventing the snapshot from being restored and after an exchange with the tech support we were able to fix the instance to have the snapshot successfully restore.
So while this is all nice and a not quite dramatic outcome, what is a bit less nice is the attitude that was displayed from prominent members after we broke the news that there was a possibility we would have to start with a blank forum. It's honestly disgusting that the initial reaction to this from some of these players was to make plans on how to get the most power and influence out of a blank slate disco. We also had people PMing staff members with informations about imaginary backups of the forums held by non-staff, people being highly disruptive with magical solutions that definitely knew the situation better than I did (don't ask me how), and of course the crazy conspiracy rumors that it was the admins and the devs behind it.
Needless to say, a fair bit of staff members are not quite happy with that. Expect some general housekeeping.
RE: Downtime - evanz - 12-19-2016
Well done Alley and Co, fantastic job, sorry you had to work all weekend on it all
RE: Downtime - Ihtyander - 12-19-2016
I will mention Your name as great benefactor of all Sirius, tommorow morning at sermon on Crete. All corsairs can join in chapel of Santa Marta to express our endless gratitude.
RE: Downtime - Croft - 12-19-2016
A bit touch and go there for a moment but bravo for getting the forums back, your efforts are greatly appreciated Alley. Again, bravo.
RE: Downtime - Backo - 12-19-2016
(12-19-2016, 12:32 PM)Alley Wrote: It's honestly disgusting that the initial reaction to this from some of these players was to make plans on how to get the most power and influence out of a blank slate disco.
I find this both hilarious and sad at the same time. I guess for some people having influence in a role playing world on a 2003 video game is really important.
That aside - good job restoring things, Alley!
RE: Downtime - Eurobeat - 12-19-2016
Just to be on the safe side: i was the only one i can think of who said anything remotely close to "it was the admins' plan" and it eas a joke. I was joking about how @Alley was leaving and taking the forum along - i think this was in the LN skype. And again, i was joking. If that was taken seriously, then i guess im sorry, but like i said. Joke
Other than that, im very happy to see the forum back online, and the staff who helped get it back deserve some big fat presents this year. Thanks guys, for working on this through the weekend and holidays.
RE: Downtime - Zelot - 12-19-2016
Great job Alley. Thanks for all your hard work.
RE: Downtime - Divine - 12-19-2016
Nice job. Was worried for a brief time but then again there was said earlier this year, there's been taken precautions to handle such situations. A bit worrying still is the "bc reasons" part, as it reads restoring hadn't quite all options available.
Nevertheless, thanks for saving our stuff.
RE: Downtime - Penpen - 12-19-2016
Thank you for the hard work, indeed.
RE: Downtime - Sombs - 12-19-2016
http://i.imgur.com/JdMtGd5.png
Looks like I wasted 13 hours of fishing in the Google Cache. However, I can't say I'm not happy about it. Kudos Alley. You did an outstanding job again.
But yeah, the behavior shown by people was really horrible. "I know a sinking ship when I see one though."
* Sombs shows middlefinger to that person.