(Resolved) GreekChat Recent Outages & Server Issues
 

For a good part of the day today (and possibly part of yesterday) GC was offline due to multiple crashed database tables. I usually check GC's status on a regular basis but found out sooner thanks to SAEalumnus emailing me about it.

Unsure exactly what caused the issue. The server did reboot around 24 hours ago so it may have been a momentary power disruption which caused crashed tables and/or corruption issues in several databases.

In any case, I spent the majority of the day & evening backing up before and after checking then repairing the database issues.

Everything seems back in order now, but if anyone notices something that is not functioning correctly with GC please let me know and I will check into it.

Thanks, John!!!!

Thank you so much!

Thanks John.

Thank you!!

Thank you so much for your work.

Thanks, John!! And, Happy New Year!

Thanks!

But in looking at all the sudden "new" user names listed (on the left in the box) on GC right now, Jon and the mods may have their hands full repairing any damage or multi-posts the spammers can muster. Good luck to our wonderful mods!

I don't post as often as I did a few years ago, but I'm still grateful for all your work on GC, John. Thank you!!!!!

John, thank you for all your hard work in keeping this site up and running. I know firsthand what a PITA database corruption is - even with a well-planned backup-and-restore plan, the restore is no picnic.

Thank you, John!

Seems we may have had a repeat of whatever caused the outage at the end of December. ZTAngel & carnation reported some errors and problems logging on to GC in the past 2 days or so.

After looking into everything and getting it all sorted out, it's basically the same as what happened 5/6 weeks ago. Server had been rebooted recently (possibly hard boot due to power disruption, although I'm not certain about it) with resulting database corruption. This time, though, GC's database wasn't so corrupted to take it completely offline but still was causing problems.

Issues have been corrected... everything should be working normally again.

Thank you John.

Thanks! I love to check GC during the day.

John .. as always, thank you for your work and support.

Thank you so much for this! I really enjoy reading Greekchat and keeping up on what's going on in the fraternity/sorority world.

Thanks again, John. I thought it was something with Chrome or Safari. You fixed it!

Poor reading skills for $500, Alex.

At first, I read the subject line as "GreekChat OUTRAGE today" not outage. :D

Thank you for getting greekchat up and running again, John. :) Happy to know the problem (in my case, for three days) was not on my end of the interwebs.

Thank you, John!

I've been away from home for a few days. Returning tomorrow and figured I would check on things here.

As my luck would have it... we had another problem with GC's server yesterday. I just finished up repairs and sorting things out. Things should be working normally again. Fortunately my tablet was up to the task.

Sorry for this additional outage over the past day.

With this becoming a regular problem, I will post again in the next few days with a plan on how to handle it.

In addition, there have been some changes/improvements that I've been wanting to make to GC for a while now and have been planning for that to begin happening very soon. This is unrelated to the recent server problems. Later this week I'll be posting more info regarding that as well.

Thank you John.

Thanks for all that you do that allows us to post.

THANK YOU for keeping our channels open.....

thanks, John. I thought it was a problem on my end b/c AZ-AlphaXi said she wasn't having any problems yesterday logging in/on. Hope you can resolve it easily without too much headache.

Ty so much! I appreciate all you do.

Thanks, John!!!!!

Server problems appear to have happened again. Was power cycled sometime in the past day (not by me). Similar to one of the previous times, some GCers are able to access the site but for others they are probably currently just seeing a database error message or something similar.

Working on this now. Should be sorted out soon.

All fixed up again. Things might be a little slow for a while, though, as I backup and download various files from the server.

John, thank you for doing all this work! I must admit that I don't know what all a forum entails.

It's not so much a forum issue as it is a server hardware issue. More specifically, database corruption issues being caused by sporadic power cycling / reboots of the server.

I'm not yet sure what's causing that to happen. Maybe I'll get lucky and it will be something on the datacenter's side of things, such as a faulty power distribution unit or similar. Otherwise, if it's the server then it could be the server's power supply unit causing reboots when it hits certain limits or maybe capacitors on the motherboard starting to go bad and causing reboots in specific circumstances. Might even be a failed/failing RAM module.

At this point my plan of action is to determine if it's on the server side, or datacenter, then proceed from there.

If it's definitely the server going haywire I'll most likely look into renting a server elsewhere and move things there for a while, rather than building a new server for GC. I'll likely build a new server for GC again at some point, but now is not the time for that.

GC's current server has been in 24/7 operation since late 2013, so we're getting close to 5 years on this hardware.

Many Thanks John!

Following are details from the server log files showing what's been going on:

Seven server hard reboots (power disruptions?) on Dec 27, Feb 1, Feb 6, Feb 7, Feb 16, Feb 17 & Feb 19.

In the quote above, the problematic reboots are in red. The reboots highlighted in blue are what they are supposed to look like. Reboots are supposed to be preceded by shutdown messages, showing a clean/graceful reboot of the server.

With the preceding shutdown log messages missing, that pretty much means the server either crashed or had a power disruption that caused the server to immediately power off and reboot.

Now for the possibly good news: The datacenter staff where I colocate the server checked on the power distribution unit after I requested, below is one of the replies I received about it:

With that, I think it's very likely the reboots were caused by the failing power distribution unit. We should be in the clear once the datacenter PDU is replaced but I'll do some additional hardware diagnostic testing afterwards to make sure.

And I say again, Thank you John for keeping the ship sailing.

Ditto! Many thanks to you John.

THANK YOU for your dedication to the good in Greek Life and your work to allow us to voice our opinions and—hopefully—offer some positive information and assistance to those who are seeking to join our ranks.

Thank you very muchly!

Happened again...

Taking GC offline for maybe 30 minutes or so to check on things.