Anyone have experience with comcast ENS?

Smoblikat

Limp Gawd
Joined
May 28, 2020
Messages
161
Hello all, I just got a chance to start configuring my new Point-to-point fiber network from comcast. I plan on migrating over to this soon-ish, but in my testing I already discovered a fairly massive issue (other than me having no experience with this sort of thing!). The general setup is as follows (actual VLANs and IPs are different, as usual):
P2P IP range - 5.5.5.5/30 (gateway to internet is 5.5.5.6 and my WAN port on the firewall is 5.5.5.7)
Site 1 - VLAN 1, IP Range 1.1.1.0/24 - Firewall, L3 switch, and main Ciena switch physically located here
Site 2 - VLAN 2, IP Range 2.2.2.0/24 - Regular ciena switch with a fully L2 internal network
Site 3 - VLAN 3, IP Range 3.3.3.0/24 - Regular ciena switch with a fully L2 internal network
Site 4 - VLAN 4, IP Range 4.4.4.0/24 - Regular ciena switch with a fully L2 internal network

Site 1 has the fortinet firewall acting as the main in/out for internet traffic, which is then connected to a Layer 3 aruba switch, which is then connected to the internal L2 network of Site 1. Port 1 of the ciena goes into the WAN port of the firewall, set the WAN port to be 5.5.5.7 with a static route of 0.0.0.0/0 5.5.5.6 gets the whole building right out to the internet (at record speeds too, im going from a 150/25 to synchronous gig!). I plug port 2 of the ciena (I was told this is my inter site port) into port 1 of the L3 switch (which is tagged to accept any VLAN from all of my sites). Site 1s LAN is connected locally to the L3 switch using a trunked pair of 1g fiber cables, the gateway address of each site (X.X.X.1) is located on the L3 switch, with a 0.0.0.0/0 static route to the internal port of the firewall. Site 1 is functioning perfectly fine on this config.

Sites 2 - 4 have no Layer 3 devices currently setup, the extremely limited help comcast would give me (apparently theyre not allowed to consult or configure customer equipment, the dude couldnt even tell me to run a ping to the new gateway....) led me to believe I can treat each site as a "layer 2" site. My understanding might be off, but if the sites are layer 2, then the Ciena switch would need my VLANs tagged on it in order to pass traffic, comcast scoffed at that notion, yet couldnt explain why. The material ive read, and the general murmorings of the internet would lead me to believe that while comcasts portion of the P2P network might be effectively layer 2, I should be able to treat it as layer 1. That is to say, a gigantic cable running between my sites with a single point of termination, to me this is the only way that I would be able to pass traffic through it without worrying about the physical switchports that my sites are plugged into on the ciena, the alternative would be that the ciena switchports come pre-tagged for VLANs 1-4094, either way this should allow me to consider this connection a layer 1 connection rather than layer 2, at least from a configuration standpoint. Would I be correct in my understanding of this, or am I completely off?

The issue I am having is that while I am able to get my site 1 moved over to the fiber connection just fine, I did a test run with site 2 where I disconnected it from its current firewall (used for the coax connection its on now) and plugged port 1 of that ciena switch into a switchport on site 2s L2 internal network, yet I cant ping any devices located at site 2 from site 1. I rebooted the stack thinking that maybe its still searching for its GW port (2.2.2.1) on the MAC address of the firewall, didnt help. I then rebooted the L3 switch at Site 1, which also didnt help. Inter VLAN routing is configured on the L3 switch, and I can ping the gateway for site 2 from the LAN on site 1. Do I need to do anything special to get site 2 to find its new gateway on the L3 switch at site 1? The way I am testing this is to run a ping to Site 2s core switch stack from site 1s LAN network, I also made a port on the L3 switch an untagged member of VLAN 2 trying to rule out any mis-tagging of the VLAN, but I still cant ping anything at site 2 from site 1. Any help would be greatly appreciated, and I can definitely provide any additional details that you need.

Thank you!
Smoblikat
 

Smoblikat

Limp Gawd
Joined
May 28, 2020
Messages
161
So just to update, I am at site 1 now, plugging port 2 of the ciena (my inter site port) into an untagged member of site 2s VLAN on the L3 switch makes site 2 work, it gets DHCP and internet. So the issue lies somwhere in my main uplink port on the L3 switch (port 1). That port is tagged for VLANs 1-4, and is untagged for the main VLAN that the uplink on the L3 switch resides on (just a /30 subnet I use to move traffic from the L3 to the firewall). I remember hearing the words "stateful VLAN" somewhere along the way here, and im trying to find some documentation on what that means. Is there anything I would need to do to get VLAN2 "ta;king" on port 1 of my L3 switch? I currently dont have any ACLs on these subnets, nor do i have any IGMP/OSPF, it "should" just be a raw L2 connection, traffic goes out untagged 2 from site 2, and does seem to get to my L3 switch as VLAN 2, but only if the port its connecting to is also an untagged member of that VLAN.

The L3 switch in question is an Aruba 2930M.
 

Smoblikat

Limp Gawd
Joined
May 28, 2020
Messages
161
SOLVED!

I found a pretty decent .pdf from aruba (totally different switch though...) that for whatever reason really helped me visualize what was going wrong. My uplink port was set to untagged VLAN2 rather than being tagged 2 and untagged another VLAN. I just made it an untagged member of the same VLAN my L3 switch -> Firewalls internal /30 connection is (VLAN 6), after that one change my laptop picked up DHCP immediately and got right out to google.

https://www.google.com/url?sa=t&rct...manual/40732&usg=AOvVaw1SWxfrhR9qhWu5gCF-1kzP
 
Top