thechronic 
admin
Registration Date: 01-11-2002
Posts: 5,293
Helpfulness rating:
 |
|
Yesterday evening the site went down. Actually the entire server went down, along with the reboot facilities. I had no access to anything and I could only wait.
There was a serious problem in the data center that caused several racks of servers to get disconnected.
Here are some excerpts of the communications that went on between the reseller that I host the server with and the data center:
| quote: |
I saw the servers pop off line I called the data center they walked out and chcked and all servers showed to be up. They then tried a ping to a couple of servers and could not ping them external or internal. They then started working on it. I called and ask them to make sure our main 3550 was up they confirmed it was up and running however the fiber leading to it appeared to be dead. THey replaced the gblic pligs that connect the fiber to our 3550 and it was still dead they then decided to try running new fiber. I called them again and told them to bypass the fiber and go to a 100 mbit copper line to get the network online and trouble shot the fiber problem later.
They have now jsut done that. But the system has to load all our IP's and config file to another router to get it to come to life. They are in the process of doing that. As soon as that is done it should it come back to life. |
| quote: |
At tis pont I cannot say for sure but it appears to me to be a issue with them getting the data to my racks.
We have a Cisco 3550 that feeds and routes all our other switches at the head of our rack that is connected to the GigE we lease from GNAX. It appears to be working, they have replaced it just in case with a new one which if it had been the problem should have taken 10 minutes to change out.
So from the time it is taking and what I know so far I would say the there an issue with them getting live fiber to us. Where and why that is an issue I am unclear on right now. When the fiber was replaced with a copper line it went live but there was a routing loop, I could ping the router but none of the servers IP's, now they have gone back to fiber it is dark again and I get no ping results at all. |
| quote: |
We've tested everything we can so far. Jeff, David, and myself are on this issue. It also seems we have a wild DOS going on which may be part of it.
When running your switch on copper 100Mbit we had nothing but routing loops giving TTL expired messages on all the pings to your servers. With fiber it's not linking up at all, we've tried hooking you up to two different routers and it's like they are rejecting the connection. We've also tried putting in a new 3550 with more or less the same results.
I assure you there's 4 of us busting our butts right now to get you back online ASAP. |
| quote: |
We ended up replacing both SX Gbics on the fiber connection...one of them died, one in the switch was already dead, and the one on our side seems to have went dead at the same time during troubleshooting. I ended up robbing a Gbic out of one of our redundant switches and got your link back up. The DOS attack that was taking place was self inflected due to your /30 dropping off the network and the routers trying to find a route to your racks. That was misleading as we thought you were getting attacked and the router was shutting the port down due to errors.
Very confusing as we have never seen this happen before.
It also appeared that the 100Mbit copper link couldn't take the traffic surge trying to get in when we attempted that setup. |
__
If you find spam on the site, please hit the  button and select my name. I'll personally kick it to the murky depths of hell where it belongs! 
|
|
