Crazy network job, corp, stories

M

Motley

2[H]4U
Joined
Mar 29, 2005
Messages
2,497
Ok everyone, lets here all your craziest stories you have related to networks at your jobs, at enterprises, etc.

Today I was thinking, whats the worst network job I've had? Hmm one certainly comes in mind.

It was about 8 years ago. I started this new job, the company manufactured/sold Soy Milk products. They were fairly small like 100 employees. Didn't have much money, they were cheap as hell.

So within my first week on the job. All the office employees were complaining their email, and connections to their servers were slow. Come to find out they had all their servers/networking equipment in an adjacent building about 500ft away. I assumed they ran a fiber cable under ground to connect the buildings right? Logic says that. Nope they had this really cheap laser solution! WTF? They had it mounted in a conference room, so that when somebody bumped it, the signal was affected. WOW alrightythen. And this happened quite a bit because they had nowhere else to mount it.

But thats not the kicker. Before I joined they had the really old dude, taking care of all network, etc. The dude was also a major pothead. LOL Yup he would take off all the time, smoke bowls and come back to work. The guy was wacky!
One day I walk into the network room, and I discover all these cat5 RJ45 splitters plugged into the network switches!!l. WTF? I asked him what heck it was for, he told me they didn't have enough drops for some new people in a conference room. When I told him you can't do that man, your gonna create a broadcast storm, overload the switch ports. He got mad at me, and I never got along with him, but luckily he was ready to retire so I didn't have to put up with him much longer.

Eventually this company was bought out by another company, had some cashflow come in, and we moved to another brand new building with an all new properly built network.
 
Don’t know that I have any crazy stories... although I am pretty jaded by the things I’ve seen. I suppose one of the craziest things I’ve seen was a company we were working with wanted to save money on equipment and switching, so in their infinite wisdom, they bought these super micro blade chassis with onboard network mezzanine cards. They then decided to create a giant loop (kinda like you would for a SAN setup with its shelves) with the first chassis uplinked to their Cisco core switching and each subsequent chassis chained to the next until the last chassis which connected to the core as well to complete the loop. Anyways, there were about 15 or so chassis and low and behold things weren’t working as expected.... lol. Anyways, for those of you who know about STP and root bridges, I’m sure you can see what went wrong here. For those that don’t, STP on Cisco is calculated to 7 hops... this is why there are generally accepted network designs already out there... like wheel and spoke.

Anyways, can’t really blame the guy too much... he’s still better than 75% of the networking folks I’ve come across. And now spanning tree isn’t as straightforward as it once was when you setup trunks between different companies using different equipment. Used to predominantly be pvst. But now, it’s mst, although you can still do pvst, but it’s just dated.
 
Kelter said:
Don’t know that I have any crazy stories... although I am pretty jaded by the things I’ve seen. I suppose one of the craziest things I’ve seen was a company we were working with wanted to save money on equipment and switching, so in their infinite wisdom, they bought these super micro blade chassis with onboard network mezzanine cards. They then decided to create a giant loop (kinda like you would for a SAN setup with its shelves) with the first chassis uplinked to their Cisco core switching and each subsequent chassis chained to the next until the last chassis which connected to the core as well to complete the loop. Anyways, there were about 15 or so chassis and low and behold things weren’t working as expected.... lol. Anyways, for those of you who know about STP and root bridges, I’m sure you can see what went wrong here. For those that don’t, STP on Cisco is calculated to 7 hops... this is why there are generally accepted network designs already out there... like wheel and spoke.

Anyways, can’t really blame the guy too much... he’s still better than 75% of the networking folks I’ve come across. And now spanning tree isn’t as straightforward as it once was when you setup trunks between different companies using different equipment. Used to predominantly be pvst. But now, it’s mst, although you can still do pvst, but it’s just dated.
Click to expand...

Wow that is crazy! 15 (cheap) switches in a loop is nuts. What was the end solution? hub and spoke?

Also a better solution now is using LACP Port Channels (vPC on the Nexus) for the uplinks, you get higher bandwidth and redundancy as well.

Yes I do see a lot of companies moving towards mst now. But where I work they are using cisco rpvst. I'd like to get some experience on mst though, so I need to push them to migrate their environment.
 
Motley said:
Wow that is crazy! 15 (cheap) switches in a loop is nuts. What was the end solution? hub and spoke?

Also a better solution now is using LACP Port Channels (vPC on the Nexus) for the uplinks, you get higher bandwidth and redundancy as well.

Yes I do see a lot of companies moving towards mst now. But where I work they are using cisco rpvst. I'd like to get some experience on mst though, so I need to push them to migrate their environment.
Click to expand...

It's been quite a few years, but I believe it was hub and spoke though. They were running some smaller catalysts to handle their L3, so pretty sure they didn't have VSS available to them and almost positive, they weren't stackable Catalysts.. definitely not Nexus, so individual trunks to each core switch.

Agree with you on a vPC if possible, but generally that is only available on more expensive Nexus equipment at the time. Today it's not so bad because you can buy some el cheapo Nexus 3ks for the price of a catalyst. Not quite sure what you mean about the LACP Port Channel in comparison with vPC though. LACP is essentially a protocol which runs over a port channel. vPC is just a virtual port channel... so you can create an LACP port channel on a vPC.

As for mst.. be careful what you ask for.. you may get it!!! lol! Always fun on new setups.. not so fun on live production environments.. :)
 
A very small non-profit asked me to design a new network. When we met to discuss my 2nd version, the manager said, "Oh, we forgot to tell you that this plan needs to accomodate an elevator, which we plan to install in 18 months." Uh... "It's not a big problem, is it? You just need to design things so the elevator can be in any one of these eight locations."
 
I think the wackiest story I ever heard was when I was in college. I had friends in a local college and spent quite a bit of time there (even used the labs all the time, haha). So one supar sharp kid from Sri Lanka that was like 16 and getting 4.0s also worked in the university support team. He was a funny dude (Shiva was his name now that I think about it), but had ZERO skills with the ladies. So he made a script that whenever a female student logged on, it would initiate a chat session with him, hahaha. He graduated with a double degree by 20 or so, still with a 4.0, and then I lost track of him. He's probably doing quite well in work, but I'm curious to know how he fared with the ladies...
 
I've been in networks (one of 2 admins) for a few years now. I went from the basics of just knowing subnetting to working with multiple routing protocols on a daily basis to troubleshoot/configure stuff across a worldwide network. The biggest surprise I've hit is IPv6. I was kind of like :rolleyes: for the longest time, until we actually had to provision a new office which I needed a /24 for and the wait at ARIN was > 2 years....and all of a sudden IPv6 is important. :)
 
Why would you need to have an entire /24 facing the internet? firewalls do nat's just fine.
 
You must log in or register to reply here.
Back
Top