Had to replace a downed switch on Christmas Eve

[Shadowspawn]

One of the Cisco 3750's in our Enterprise stack died last night and I am the only one on call this week, requiring me to drive back in and replace it. I hate that damned stack - seriously needs to be upgraded. They are the old 1.5u models with little ram.

Anybody else have a holiday emergency?

 

[Jay_2]

No, I'm on call from 12am until 7am tomorrow then in the office from 7am until 3pm. Same all week :-(

 

[Ruoh]

I checked all the logs before I left work on Monday. Blew some air through the rack on a couple of the switches for good measure... All was good.

 

[Jay_2]

I checked all the logs before I left work on Monday. Blew some air through the rack on a couple of the switches for good measure... All was good.

We have 4 Datacenters, it would take me about a week to do that

We actually lost a blade today, noticed the alert, HA sorted that out for me so I can wait until tomorrow morning.

 

[Ruoh]

We have 4 Datacenters, it would take me about a week to do that

We actually lost a blade today, noticed the alert, HA sorted that out for me so I can wait until tomorrow morning.

Sometimes being small is better.

 

[MysticRyuujin]

If it was a redundant stack why couldn't it wait? lol

 

[Red Squirrel]

Thankfully nothing eventful was going on this year.

last year there was storms throughout parts of Quebec and we had tons of COs/cabinets/cell sites on generator power and NOC was running on 2 guys even off regular hours just to keep up with coordinating techs throughout the region. only dropped a few small sites but overall managed to keep mostly everything operational through the massive outages and ice storms. Was pretty crazy. For us it was not that bad but for the CO techs they pretty much had to work throughout Christmas. Sucked for them, though I'm sure it was good money.

 

[jeffmoss26]

Internet was down for some period of time, got a text at 6 this morning and a couple VPN users emailed me. I rebooted the firewall when I got in and everything seems okay. We were closed Tuesday and Wednesday.

 

[0V3RC10CK3D]

What firewalls do you run? We've been using Juniper at about 100 locations and have yet to see a single firewall related issue in 10 years, most of these are the cheaper SSG5 units too

 

[Shockey]

If it was a redundant stack why couldn't it wait? lol

Was thinking this too. We built/designed it for uptime and so we don't get called in middle of the night

Maybe it was impacting performance?

 

[MrGuvernment]

wow, co-incident, our ISP had a distribution switch that took out 32 clients go down x-mas eve about 10pm -6 time., and god knows why no fail over.. they had to manually change the switch and reconfigure apparently.. i assume because no one does config backups....

What firewalls do you run? We've been using Juniper at about 100 locations and have yet to see a single firewall related issue in 10 years, most of these are the cheaper SSG5 units too

Any suggestions for a Juniper FW for a network that has about an avg of 45k states at any given time....since it seems most of their cap out at 50k sessions, or am i confusing states and sessions...

 

[calvinj]

Both Air Handlers in our server room needed recharged. Ended up getting 98+ in there for a while. Only thing saving us was being below zero temps. We opened a few windows on the floor and the big fans in the door did the rest. Was able to get it down to 94-95 while the AC guys trying to get it all figured out

 

[Shadowspawn]

If it was a redundant stack why couldn't it wait? lol

Was this question aimed at me? </me checks original post> Nope no mention of redundancy.

Actually, it is a redundant stack, but not fully redundant. Most of the VMs running on that stack of switches can switch to another stack in the other building, but there are a couple that cannot for whatever reason.

No matter, it was a quick swap, no programming needed. Just swap the physical switch. It was just annoying to be in my sweats and have to throw on some clothes and drive 30 minutes each way.

 

[jeffmoss26]

What firewalls do you run? We've been using Juniper at about 100 locations and have yet to see a single firewall related issue in 10 years, most of these are the cheaper SSG5 units too

Our core firewall is a WatchGuard Firebox X1250e, we have smaller units at 2 branch offices.

 

[MrGuvernment]

we had an x500e, while it worked i found it very cumbersome on the interface and configs, mind you this was...8 years ago..

How are they now, in relation to Juniper and Cisco for FW / Routers, since Cisco does the support fee's like WatchGuard does...

 

[jeffmoss26]

WatchGuard is the only firewall I've had much working experience with...it can be a bit painful at times. Certain configuration settings are not where you might logically think they would be.
I have not had to call for support but I am sure my boss has over the years.

 

[Jay_2]

I have used watchguard... really poor IMO bad interface and software. It was about 3 years ago now though.

 

[MrGuvernment]

Ya i hated our watchguard box for configuration, it is like the GUI and interface applications were purely designed by developers, not users.

 

[QHalo]

If it was a redundant stack why couldn't it wait? lol

Loss of a switch in a stack still means you have to go swap it. I wouldn't run in a degraded redundancy state just because I didn't want to go into the office.