Office Network Questions/Sounding Board

Hey Guys/Gals. So my office runs a 24 port HP Gigabyte Switch that was hooked up to a Watchguard firewall. Our IT provider came down yesterday because we are upgrading our service provider and part of that upgrade is the provider is taking over the firewall on their end. It also turns out our current cable modem is broken and only providing 5% of our total speed, our cable provider is coming out to replace it at their leisure (as usual).

Now, our IT rearranged all the ethernet cables as the watchguard was coming out, and now half our office doesn't have internet. They are blaming poor terminations on the Cat6 cables as well as needing a new switch. So I have a few questions and need a bit of a sounding board.

Additional information:
1. I have confirmed that the cables are working from Phones/PC's to the switch, as in both sides light up when plugged in and turn off when unplugged.
2. I have also confirmed that the terminations are not great, as some cables can be wiggled and they lose connection briefly.

Questions:

1. Does the sudden loss of connection add up to poor termination?
2. Is the switch involved at all other than being old?
3. Does this sound more like a cable modem problem?
4. Did they f up?

thank you in advance.

I have also now tried fresh cat 6 cables with no luck to either the modem or the switch.

Feels like someone is f’ing with me.

What did the 'rearrangement' consist of? If they helpfully pulled the Watchguard out and it was the DHCP server, could just be a simple case of expiring leases with no renewal option. Might check one of the no internet workstations for the dreaded 169.254.x.x addresses.

Unlikely the switch is to blame but you could test by swapping some of the non internet PCs to a port that has a working PC. Switches can fail in weird ways.

Doubtful poor termination is the cause of the sudden loss unless the rearrangement broke a bunch of connections.

Answer to question 4 is YES as they changed crap, broke connectivity, and left without fixing it or putting it back the way it was.

It was working before they changed things and not working afterwards. Definitive fuck up!

As previously stated the Watchguard was likely providing DHCP. Whatever was put back could be handing out address from a different network. You should compare the IP+mask from a workstation that has Internet access to one that does not. The correct procedure should have been to reboot all client devices after the install or to at least have put a secondary IP on the inside interface of the new router matching the old. Based on your story I'd be looking for a new service provider. They should not have left before checking that connectivity had been restored.

He did stick around for a bit, got half of things working then left saying we where lucky to get his time and couldn’t afford his hourly. I was in a meeting or I would have told him you break it you buy it.

Basically our IP phone system is borked, so are our IP scanner/printers, half the pcs work with cables, the other half connect to the wifi atm. Everything was cables and working before. He also didn’t snap a photo of the prior setup before rearranging.

Then his office sent a PO for a 7k switch.

The kicker is I can plug phones into the cable modem and still get nothing. I think it was the watch guard like you said.

I settled on sending him packing this afternoon, just need to either unbork this myself or find someone else.

$7K for a switch? What's that office smoking?

Not a recommendation but just one of many that popped up on a search for 48 port gigabyte switch: https://www.amazon.com/Cisco-Busine...1734&s=gateway&sourceid=Mozilla-search&sr=8-5

$500 and many similar for similar prices. I used 48 port to allow for future growth.

Based on what has happened so far, I would ask a LOT of questions on the type of firewall services the new provider plans on providing. You could find yourself with the basic consumer grade non-firewall that will leave you open to easy malware attacks.

Might try connecting one of the LAN ports of the Watchguard to your switch and see if things start getting IP addresses.

Dead Parrot said:

$7K for a switch? What's that office smoking?

Not a recommendation but just one of many that popped up on a search for 48 port gigabyte switch: https://www.amazon.com/Cisco-Busine...1734&s=gateway&sourceid=Mozilla-search&sr=8-5

$500 and many similar for similar prices. I used 48 port to allow for future growth.

Based on what has happened so far, I would ask a LOT of questions on the type of firewall services the new provider plans on providing. You could find yourself with the basic consumer grade non-firewall that will leave you open to easy malware attacks.

Might try connecting one of the LAN ports of the Watchguard to your switch and see if things start getting IP addresses.

Click to expand...

Tried that already, is there any specific port and/or does the firewall need to connect to the modem then to the switch?

My network IT skills suck, or I would do it myself like everything else.

As for price, yeah I figured that one out with a quick google. They had some 3 level cloud managed switch for a 10 person office that only plans to grow to 15 (I had 40 in a prior life and it was an absolute nightmare managing staff I got nothing else done). Planning to grab a $500 switch and a new buddy of mine works in cyberspace security so I’ll give him a look over the system.

Pretty sure their smoking the year end bonus special coupled with a make work for themselves whiskey.

The SOB left knowing things were bjorked up!? FTN! They need to come back and roll back to the last working state then you fire them. Assuming this is simple layer 2 switch is there anything between the cable modem and the switch?

Yes, certain ports on the Watchguard will need to plugged into certain places.

Assuming this is simple layer 2 switch is there anything between the cable modem and the switch? Do the boxes that can't get to the Internet have IPs, routes and nameservers? What are they and are they different on the boxes that can access the Internet?

Just to hand out IP addresses, you would connect one of the LAN ports to the switch. After a bit of thought, if the watchguard is the DHCP server, it is likely also setup as the gateway. If so, you will need to also connect the WAN port to the modem. Making things much like they were before your reconfig. This assumes you still have the original modem and ISP provider at the moment. Might at least let you get to the weekend.

It is possible the modem is handing out some address. Might be it wasn't setup to have a large enough scope as they expected the watchguard to handle that and NAT one address from the modem into many for the office. Many possibilities.

Assuming Windows PCs, get a command prompt box open and type ipconfig
You might have to scroll up to see the results but look for a heading of Ethernet adapter Local Area Connection
Should see an IPv4 address of 192.168.x.x or 172.16.x.x or maybe even a 10.x.x.x type address.
If you see 169.254.x.x then you aren't getting a useful address.

Also note the gateway address.

If the watchguard is the DHCP server and the gateway, then the gateway address on the PC has to match the LAN IP address of the watchguard.

Yeah, he left with bits of the system working, claiming it was our cat 6 terminations, and the switch and the cable modem. Also that none of this was his responsibility (his firm did our cat6 terminations in 2016)

Pretty pissed atm because he stopped by to fix a small issue with one of our staff systems and decided to muck about with the firewall because our new business connection has a built in firewall. Now I have to review the entire system and am considering using a it security systems consultant.

Basically wasted two days I could have been doing money making work.

And the wifi access points are down this morning, those are part of our ISPs system.

It's just computers and phones and wifi points all connecting to the internet? You don't have any on premises servers?

Sulphademus said:

It's just computers and phones and wifi points all connecting to the internet? You don't have any on premises servers?

Click to expand...

There is a server here as well, tracking down the problems now, replaced the old 24 port switch with a Ubi 48 port EdgeSwitch, seems to have hammered out some of the problems.

I didn't check the server for connectivity, I just assumed it was down with everything else.

Dead Parrot said:

Answer to question 4 is YES as they changed crap, broke connectivity, and left without fixing it or putting it back the way it was.

Click to expand...

As someone that works in IT I agree here. Regardless of what the problem is, it was working before hand and it was left broken after they messed with it. They should have fixed what they broke.
Sucks how almost no one in IT is able to take ownership of a problem they make.Oh the blame game... it really sucks.

dany man said:

As someone that works in IT I agree here. Regardless of what the problem is, it was working before hand and it was broken after they left. They would have fixed what they broke.
Sucks how almost no one in IT is able to take ownership of a problem.

Click to expand...

That so much, when I was told he blamed everything else but his work all I could say was 'typical it guy'.

Aireoth said:

That so much, when I was told he blamed everything else but his work all I could say was 'typical it guy'.

Click to expand...

I know, I see it all the time when dealing with ISPs.
We needed to close the service at a site and it took 2 days of screaming on the phone before they closed it because they changed the names on the account.
I was told they did not want the equipment back. Lucky I saved it knowing this was a lie. A month latter a repo guy got sent out to the old site that we no longer owned despite being told we sold the site.

TBH 95% of the people in IT and networking need to get the boot.

Ok, so I put in the new switch, WIFI works, but if I move any ethernet cords I have to reset the switch or they are forever resolving. I know enough to know its a software/CLI thing, but not how to fix it. At least I limped along since Wednesday to have a mostly working office at this point.

Off to find someone to take this job over from me.

As for hardware, amusingly enough our ISP's seem to be the opposite, I have never had them come back for their stuff except on my home line. I still have left over ISP equipment from an IT firm that inhabited this building a decade ago.

Is it a managed switch? It may need configuration as well.

Brian_B said:

Is it a managed switch? It may need configuration as well.

Click to expand...

Yes it is, but every time I've work on the software site of networking it takes me forever.

I'd have to figure out what needs to be managed, how to do so, what information need to be completed to manage it, and often check, recheck, double the recheck and hope it all works.

I would expand the scope of your proposed cyber security contract to include an appraisal of your network needs. I wouldn't trust anything your current(now former?) IT folks have said or provided.

Since you have the Edge Switch, assuming that one does POE, you might look at switching your wifi APs over to the Ubiquti versions. They supposedly have a pretty nice does it all management software package. They also sell a firewall/router appliance if you wind up needing one of those. It would limit the "its THAT vendor's fault" finger pointing if a gizmo stops working.

One last thing.

Is the Watchguard firewall able to resolve or be forgiving when it come to wiring imperfection/speeds compared to an ISP's modem (this one is a Hitron)?

Edit: Nevermind, I answered my own question. Thanks for all the fish!

Aireoth, is everything straightened out now? or are you still having issues?

You've already ruled this out by replacing your switch, but that MAY have not needed to be done.

Many people in the Networking space don't pay enough attention to layer 2 issues. In the old days of networking it didn't matter much of what port on the switch that you were plugged into(this is still true of layer 2 switches) however with layer 3 switches and new security features it can matter a great deal.

One thing I always try to remember to do when rearranging cables on a switch is to clear out the arp table, "clear mac address port all"(i believe that is the command for HP), either that or reboot the switch and that will have the same effect. But there is a caveat.....port security.

Port security is great, is blocks someone from just being able to unplug a working office PC and plugging in their own laptop. It will detect it's not the office computer and shut-down that port helping protect your network from unauthorized computers.

When people normally enable this they also include the command, "MAC address sticky"(on Cisco) what this does is basically put a static ARP entry in the MAC Address table(how the switch knows what port to send information too) , Static ARP entries take precedence over dynamic ARP entries, so if someone randomly unplugged everything and plugged everything back in, only if he got lucky and plugged the same machines back into the same ports they were on previously would it work. It would almost seem like just random computers/phones were working when in reality as long as he plugged the same machine into the same exact port as before would it work.

Just something to check for next time.

probably at this point if you havent done so log into the switch and just open all the ports and put all of the ports on the same vlan. sounds like you have a small office so just have the segmentation take place at layer 3, you wont have storm issues or anything like that unless you have a device physically connected to every port on the switch, but sounds like the bulk of users are wifi workstations, and that is working.

you wont be able to figure this out yourself if there are no backups or some kind of documentation on how things should be working. you need to call a it provider for some dedicated network support because some rearranging is going to happen to fix this i bet.

as for him peacing not defending his attitude at all BUT BUT BUT if a cable is messed with after sitting there for years the copper WILL break and connectivity for that wire will be down. ive had plenty of ancient cables go half duplex after some rearranging with no logical changes.

edit: oh this is a week old now. how did you resolve?