server crash

i just tried to get to sosh and it looks like the server is down, i can't connect to it from the network side. assuming network issues at colocation provider, opening ticket

nip

CHB

Curly Haired Boyfriend

We love our Curly Haired Boyfriend

baseball news

baseball news

mlb news

Fake Rolex

Fake Rolex

soshdown

from our host

==================

At this time, we are still waiting on the last 30 or so servers to be manually rebooted- we originally had a list of approximately 70 servers (about 2.30 hours ago) and while some servers only needed a manual reboot and will be back online a few minutes after the reboot is issued, other servers may take longer if after the reboot was issued, they require a complete file system check. For this reason, any eta that is given is an estimate based on overall progress of all restoration.

Regarding the process of repair after an outage such as this: It was found that a large number of the servers at the FITX datacenter were unreachable this morning (at 7.30am EST- we are still trying to get the exact reasons for this). Network and hardware technicians at the datacenter on-hand were dispatched to ascertain what the issue was and correct as soon as they were able to minimize the amount of time servers are offline. Roughly 2 - 2.30 hours after the initial outage began, most of the servers that were affected came back online. During this time, additional technicians were called in to assist with the restoration of those that were still offline.

As soon as the network came back online, we compiled a listing of the servers that had yet to be restored and provided it to the hardware/network technicians to begin checking each one and manually reboot as needed. While this process will ensure that each server still offline will be brought back, it is a slower phase of server/network restoration since it requires that the techs go to each rack location, reboot as needed and verify that the server comes back online before going to the next server.

Our technical support staff are doing their best to follow up with the hardware technicians about progress and are also watching our monitoring software for a rough idea of how quickly the remaining servers are coming back online so they can respond to tickets and chats accordingly. As such, the etas they are providing are generalized and may fluctuate depending on how long each server needs- for those that do not come back right away, fscks may need to be performed and this can slow down progress. That said, we are doing our best to restore access to the remaining servers as quickly as we are able and the number of servers still offline has dropped by 5 or more since I began writing this.

We will provide updates as soon as we have them regarding the cause of the initial outage.

nip

8:45pm

my most recent post with our provider

==============================================

we are still down and, as usual, i can't get anything but "we're trying out best"

WE HAVE BEEN DOWN FOR EIGHT HOURS NOW CAN WE GET SOMEONE WITH SOME GODDAMNED ACCOUNTIBILITY INVOLVED?!

Mark, you said you would "update here with any additional information" that you get. Can I assume that we've been down for an additional three hours and there is literally no more information available? I could have driven there and rebooted the fucking thing myself by now.

7:30pm

we are one of a dozen sites with this host still down. I have parsed out a few choice posts , but believe me our provider knows how pissed we are.... anyhow here is the director of sales most recent communication

======================
OK we still have about a dozen or so down and are working towards getting those last ones up.

I have no further update beyond that and really do not want one. The time it takes to reply to those questions can be better spent on servers and getting them back up...That has always been my philosophy and always will be.

Once this is over, I will post here what happened, etc...but I have posted all I really need to know....The power failed, the UPS, due to a known but as yet un-repaired issue automatically cross over and so power was denied to the servers for a period of time...After that mot all came back right away and needed manual intervention.

I fully agree this has been unacceptable, I have voiced grave concerns not more than two weeks ago to the senior staff at FITX and I know a number of changes, upgrades and improvements are slated in the very very near future (including the UPS repair tomorrow)....

I will update with any further information I have.

===================================

6pm

we're still down, and for three hours we've been waiting for them to power cycle us. Truly a shitty host.

3:50pm

we're still down. our host's message board reports that some sites started popping up recently, I expect us to be up shortly.

2:33pm

while i have your attention, feel free to friend us at

http://www.myspace.com/sonsofsamhorn

network problems still causing pain. server is safe. rejoice.

7/28/2006 2pm

confirmed network issues at the data center that our server lives in

7/28/2006

1:48pm

just got through to support. eta en route

7/28/2006

i'm trying to contact our hosts but they appear unreachable. I suspect a network issue.

7/28/2006

just got back from lunch and see sosh is down... investigating.

12pm update

ok, the drive corrupted our database, fortunatelly we do have backups. Unfortunatelly they are on a drive not attached to the machine. Our favorite tech is on it, it might be a few more hours. I will not know as I'm going to try to catch some sleep. I have contacted some dopes/mods and one of them will pick the issue to make sure it gets worked on through the night.

11pm

the board may look up but it's not - tables are stil seriously corrupt. we may have to fallback

williams head case, it was nice having you here by the way

10pm

they're still restoring... the issue remains that there was a bad disk and the restoration process has had a few setbacks. we could restore from backup.... but we've been down for long enough so I figure it's worth it to atleast attempt to do it right.

9pm

they're still working on things, just spoke with them 15 minutes ago.

no, i'm not making excuses for them... they suck but the unfortunate reality is we're stuck with them atleast in the short-term

5:20pm

Comment from our hosting provider

Need 1 to 2 more hours...database is HUGE

4pm update

all software is updated and final tweaks are being made. this is where we see how our database fared through all of this, so this could take 5 minutes or 5 hours.

While we do that, MySpace users feel free to join the SoSH group there

http://tinyurl.com/mwb2e

3:30pm

rebuild still underway.

- nip

why not?

sosh is down, my hands are tied, here's some boobies to tide you over!

Model Brittany Brower, poses with a Sidekick 3 as she arrives for the T-Mobile Sidekick 3 launch party in Los Angeles, Tuesday, June 20, 2006. (AP Photo/Lucas Jackson)

quick request

if you know folks who don't know about this blog then please point them in this direction as this is the only place that updates will be made available.

just got off a follow-up call with the salesfolks at our host. They have installed the new hard disk and are aiting for the OS/data reimage. This should take approximately 2 hours. A very fair eta is 5pm edt, but i'm guessing we beat that

12:30 update

confirmed it is a bad disk. instead of these off/on problems i am having them replace all of the hardware. eta to get back up is before 5pm

- nip

10am

Most recent and very full update

Last night the server went down in typical fashion. Over the past few weeks we have been isolating the cause of these outages. Each outage we're replaing a new piece of equipment.

This most recent crash was hard on our database, I believe the actual disk our database is living on is corrupt. We ar ecurrently awaiting our host to replace the motherboard on our server. They are taking their time -- bad for them as it's giving us opportunity to investigate contingencies (i.e. move to another provider).

I'd love to say t'these things happen' but they really shouldn't. Upon moving away from ezboard we were running on the latest and greatest machine and did not think it could go south this quick. Our upgrade designs which were going to be implemented in the offseason are obviously being accelerated. We hope to complete the upgrade, which is more of a total overhaul, within the month. This will include geographic diversity (which would have taken care of the ddos attacks from a few weeks ago)

again, we apologize. we're all putting off our real lives and real jobs a smuc has is feasible to get this up and running. my wife is pissed! ;)

nip

Still Down 8am

There are hardware issues that have to be fixed. No ETA for SoSH to be up.

-Brandon