Login

Username:


Password:


Remember me



Forgot Password?




 Merchandise




Site downtime

Please post help requests in the Helpdesk forum. This forum is to be used for discussing the site, and events/projects.

Site downtime

Postby Spack » Wed Aug 12, 2009 10:45 pm

Due to an unfortunate accident the power was out in our building for a while. I'm was hoping to move the server to the new building tomorrow but that looks like it might have to wait till Friday now as I've spent all day trying to get the company network up - the power spike from the outage blew both our Cisco PIX firewalls (1 completely dead, the other damaged flash ROM and so wouldn't boot). It's almost 11pm and I've spent the past 7 hours learning how to configure a Cisco ASA 5505, looks like it's working now :)
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby mattjgilbert » Thu Aug 13, 2009 8:59 am

Good work Mr Dan.
User avatar
mattjgilbert
BladeDancer
Daemon Prince
 
Posts: 5847
Joined: Wed Sep 19, 2007 12:00 am
Location: Worthing, UK

Re: Site downtime

Postby Spack » Thu Aug 13, 2009 12:28 pm

Got another ASA 5505 arriving tomorrow, so we'll have a full failover config again :D
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby conscriptboris » Thu Aug 13, 2009 10:35 pm

:O

didnt you have any magnetic surge protectors??!?!?!?!?!? and UPS cabs?????

I have these on all our equipment :D

Elvis
User avatar
conscriptboris
Elvis isnt Dead!
Commander
 
Posts: 1558
Joined: Thu Jan 24, 2008 12:00 am
Location: Gibraltar
Medals: 1
Painting Entrant (1)
Blog: View Blog (1)

Re: Site downtime

Postby Spack » Fri Aug 14, 2009 7:32 am

It was the UPS that tripped (we have a single UPS for the entire comms room). I'm beginning to think that as nothing else was damaged it wasn't a surge - it was simply the loss of power that knackered them. They may well have not powered back up had I removed them and moved them to the new office either. These firewalls had been running 24/7 for the past 9 years, they may well have just been on the verge of failing and the power outage took them over the edge.

The new network is going to be more resilient - the two ASA firewalls are going to be in entirely different rooms with gigabit ethernet connecting them for the stateful data link. We couldn't do this with the old ones because they had a special failover connector which only had a 30cm cable so both units had to be one above the other in the comms rack.

Our fileserver was the most fortunate piece of kit though - one of it's power supplies was on our small rack mount UPS for our new office cabinet (I was using it to test the UPS as it was on my desk anyway being configured, and has 2 power supplies), so it didn't go down. Not a lot of use without the rest of the network though ...


There will, if all goes to plan, be downtime today - I'm taking the AoS server to the new office. I'll be leaving a reverse proxy in place so while the DNS updates are going through it should appear to be a transparent transition (other than the 20 odd minutes it will take to drive the server to the office and plug it in!).
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby killmaimburn » Fri Aug 14, 2009 7:53 am

Remember to leave some kind of OUT OF THE OFFICE BACK SOON message rather than just a 404 message (or does that require some very complex procedure?)
Barely even lurking..
ruffian4 wrote:Handy fellow, this kmb...Like Ahriman delving the paths of the webway ...
World of ME First try at Apoc Batrep WHAT/WHO is AOS?
User avatar
killmaimburn
Now Vanus Clade
 
Posts: 6581
Joined: Fri Sep 21, 2007 12:00 am
Location: Nottingham, mid-land
Blog: View Blog (1)

Re: Site downtime

Postby Spack » Fri Aug 14, 2009 8:15 am

Yeah, I will.
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby mattjgilbert » Fri Aug 14, 2009 8:47 am

I've had a UPS burn out on us before in a previous job.
User avatar
mattjgilbert
BladeDancer
Daemon Prince
 
Posts: 5847
Joined: Wed Sep 19, 2007 12:00 am
Location: Worthing, UK

Re: Site downtime

Postby Spack » Fri Aug 14, 2009 12:13 pm

This one was only serviced last week. As it is though, we can't prove it was a fault with the UPS, and the engineer who came out to check it could find nothing wrong with it.
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby Spack » Fri Aug 14, 2009 7:12 pm

Sorry about the long time down, was trying to get it to work at the new office and it just wouldn't work. I could ping out but nothing else was working at all. I've returned the server back to it's original location, and I'll be trying again on Monday.
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby Spack » Tue Aug 18, 2009 7:23 pm

The move is back on! It should be moved tomorrow, all being well. I've got to put the internals in the new 4u rack case that arrived this morning, and then get it into the server rack in the new office. I'm aiming for no more than 1 hour of downtime, although I'm pretty sure something will go wrong so expect 2 :P
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby conscriptboris » Tue Aug 18, 2009 7:30 pm

As always, anytime you move something it will fail for no reason.

We had to move a frame room the other Xmas (due to it not being used much at that time, oh joy) it involved 6 frames (copper) and 2 48U cabs Fibre and 3 48U equipment cabs and a 3 stack siemens exchange.

and we had multiple systems not boot up correctly and these were things like switches and routers, nothing to special???, well ol Q. Elizabeth bought some new ones :D

Also, wierd as hell one, we had a media converter (high bandwidth) coax to fibre that had worked since i had been there (one of the external access points for the exchanges onto the civilian lines) once it had moved 20m, it would work?

Upon further inspection we had it running over 2km of fibre, and the local telecoms had it running over 5km!!!?!??! of fibre. The quality out here is shit so it stopped working, that took me 8 hrs to figure out (the converter was specific 48hrs dispatch so had to keep working). Didnt help that gibtel kept telling me it was only 1km.....bastards,

anyway rant over, hope all goes smoothly!

Elvis
User avatar
conscriptboris
Elvis isnt Dead!
Commander
 
Posts: 1558
Joined: Thu Jan 24, 2008 12:00 am
Location: Gibraltar
Medals: 1
Painting Entrant (1)
Blog: View Blog (1)

Re: Site downtime

Postby Spack » Tue Aug 18, 2009 7:50 pm

We lost (ie dead!) one PC today in the move - unfortunately it was the PC that interfaces our Sage system to all our suppliers! Luckily I'd made a copy of the applications and order files as it was due to be migrated to one our new virtualised servers, I just hadn't finished testing it. I had no choice but to commission the new virtual server there and then to get the order processes running. Apart from that, it's gone pretty smoothly so far - 2 critical servers moved and all the office desktop PCs. Tomorrow it's the web server and new mail/image server along with the AoS server, hopefully they'll go nice and smoothly. I'll leave the reverse proxy in place about a week along with the first ASA firewall, then it'll all get shut down and I'll drop the ASA in next to the one in the new office as a failover unit.

I did have a bit of trouble with segmenting the switches in the comms room and office into 3 VLans, as I had only 1 gigabit connection between them and so packets were picking up on the wrong interfaces by the firewall. Ended up figuring out I had to switch from a port based VLan to an IEEE one with tagged packets, and now it works great.

I'll take more photos when the kit is all in place. It's a lot neater than our current office :)
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Re: Site downtime

Postby conscriptboris » Tue Aug 18, 2009 8:00 pm

OOooohh, neat and shiny, this is what i love about the IT/Comms world, no one realises the effort or work you put in, until disaster strikes and then they are sitting on your shoulder!

The amount of times ive implemented cost saving and performance boosting procedures and never got so much as a thank you, then when something blows up (because they didnt want to spend that £3000) and I sort it out, then im the hero all of a sudden.

I actually had trouble with some of our routers this week, installed a new cisco router in chain with a 3com hub, the collisions went throu the roof! had to set the cisco to be a slave(?), so basically i had a really expensive hub....

Elvis
User avatar
conscriptboris
Elvis isnt Dead!
Commander
 
Posts: 1558
Joined: Thu Jan 24, 2008 12:00 am
Location: Gibraltar
Medals: 1
Painting Entrant (1)
Blog: View Blog (1)

Re: Site downtime

Postby Spack » Wed Aug 19, 2009 2:40 pm

The site has been moved, proxies are in place, hopefully everything will settle down over the next couple of days :)
User avatar
Spack
I R Ginger
Field Marshall
 
Posts: 6779
Joined: Sat Feb 12, 2005 12:00 am
Location: Birmingham, UK
Medals: 2
Bronze (1) Painting Entrant (1)
Blog: View Blog (29)

Next



Return to The Site




 Social Links