IPSec + BGP on PFSense (mixed policy and routed IPSec)

L

lopoetve

Extremely [H]
Joined
Oct 11, 2001
Messages
33,891
Sanity checking a few things here. Goal is IPSec that learns from BGP, since each site has subnets that are created/destroyed regularly (routed), and talk via BGP to update routing to the core.

2 sites. Site A, Site D (there will be a B/C eventually). PFSense handling core routing for both sites (2.5.0). IPSec tunnel with policy based routing configured (2 VLAN/Subnet per side are hard coded - management, and hte BGP Transit lan where the other routers talk to the PFSense core router).
EG Subnets at Site A:
10.10.1.X/24 - Management
10.10.11.X/24 - BGP to PFSense connection, basically .1 for PFsense, and .2/.3 for the next router (dual homed, redundant) down right now -

Current config has the one subnet for the DR side (hence site D) linked to both hard-coded VLANs at site A (ignore .2, it just went away - that doesn't need to route).
1619034751855.png


This works fine. On the BGP side, the next router down (T0 router, vs TNull for PFSense) is working perfectly, and the T1 routers/subnets created below are working fine:
1619034854187.png


20.0 and 21.0 are both On a T1 - goes T1->T0->PFSense (the lower layers are NSX-T, if you care).

Now - I want those routes to propagate across to site D, which only has the first set (1/2/11) accessible.

My understanding - I need to configure an IPSec VTI - I'll use 10.254.253.0/24 for this, because we're lazy and use /24 for everything, and put SiteA on .1. I'll also configure a .2 for SiteD, and... something then? That's where I get confused a bit.

Will be trying this shortly. Expect questions to follow when this goes weird.
 
Confused on your local/remote subnets - they're completely different. They should be the same subnet as each VTI interface needs an IP on the same subnet to establish the BGP peering.

I have a very similar setup between my family members homes. My parents house is the "core" with tunnels going back to my place and my sisters place. Only exception is no BGP...as these routes/subnets will never change. (unless I change them :) )

My P2 settings:

1619763814254.png


Mode is showing VTI, not tunnel.

1619763852935.png


Local/Remote networks are set to "address" pointing to the remote interface.

You then need to create a new interface under interface assignments and map it to the IPSec VTI tunnel you just created. Config is fairly straightforward.

1619764015618.png


Make sure bogon blocking is unchecked for obvious reasons. Don't forget to add a firewall rule under the "IPSec" tab allowing all traffic.

Once this is done on both sides and the tunnel comes up, you should be able to ping each VTI end point by logging into the shell:

1619764205588.png


At this point, you are ready to setup routing. As I said before, I'm running a static setup so I just have static routes setup in each pfsense boxes table...and I can actually hop via the central pfsense box (star topology) to actually reach the other endpoint networks if I want to.

Hope this makes sense.
 
ComputerBox34 said:
Confused on your local/remote subnets - they're completely different. They should be the same subnet as each VTI interface needs an IP on the same subnet to establish the BGP peering.
Click to expand...

Correct - at the moment, I'm not yet using VTI, we're using policy-based routing (for the core VLANs, to get things up and running between sites/etc). We're going to switch over to VTI (since BGP sees the policy based links, it gets confused if you try to use both at once :)). Our transit lan will be 172.16.254.0/24, which is overkill, but I hate anything smaller than a /24.
So that's coming as soon as I have someone on that side to do things in case we break it, so that we can revert back to the policy based routes.
ComputerBox34 said:
I have a very similar setup between my family members homes. My parents house is the "core" with tunnels going back to my place and my sisters place. Only exception is no BGP...as these routes/subnets will never change. (unless I change them :) )

My P2 settings:

View attachment 351952

Mode is showing VTI, not tunnel.

View attachment 351953

Local/Remote networks are set to "address" pointing to the remote interface.

You then need to create a new interface under interface assignments and map it to the IPSec VTI tunnel you just created. Config is fairly straightforward.

View attachment 351954
Click to expand...
Yep - this is the moment the cross-site link dies on the policy portion, so I have to do this logged in locally with an OpenVPN link back to the second "site" to make the changes there - and I want someone there in case we totally break it to restore the saved prior configuration.
ComputerBox34 said:
Make sure bogon blocking is unchecked for obvious reasons. Don't forget to add a firewall rule under the "IPSec" tab allowing all traffic.

Once this is done on both sides and the tunnel comes up, you should be able to ping each VTI end point by logging into the shell:

View attachment 351956

At this point, you are ready to setup routing. As I said before, I'm running a static setup so I just have static routes setup in each pfsense boxes table...and I can actually hop via the central pfsense box (star topology) to actually reach the other endpoint networks if I want to.

Hope this makes sense.
Click to expand...
I'd do the star, but with 4 full sites doing stuff, running everything through the core would be less-than-fun.

Theory: If I have 172.16.254.0/24 as the VTI transit lan, I can have site 1 be .1, site 2 be .2, and so on - and BGP peer across those 4 addresses, right?
 
lopoetve said:
Correct - at the moment, I'm not yet using VTI, we're using policy-based routing (for the core VLANs, to get things up and running between sites/etc). We're going to switch over to VTI (since BGP sees the policy based links, it gets confused if you try to use both at once :)). Our transit lan will be 172.16.254.0/24, which is overkill, but I hate anything smaller than a /24.
So that's coming as soon as I have someone on that side to do things in case we break it, so that we can revert back to the policy based routes.

Yep - this is the moment the cross-site link dies on the policy portion, so I have to do this logged in locally with an OpenVPN link back to the second "site" to make the changes there - and I want someone there in case we totally break it to restore the saved prior configuration.

I'd do the star, but with 4 full sites doing stuff, running everything through the core would be less-than-fun.

Theory: If I have 172.16.254.0/24 as the VTI transit lan, I can have site 1 be .1, site 2 be .2, and so on - and BGP peer across those 4 addresses, right?
Click to expand...
I have never tried to "stretch" layer 2 across multiple VTI interfaces like that. When I've run this design in production with my Cisco gear, I always needed a separate subnet per tunnel per connection. In your case, I think that would be 6 tunnels and therefore 6 distinct subnets. With the way PFSense is configured, it does not really give you the option to specify a mask. As soon as you select "address" above "type," it basically assumes a /30 or maybe even /31 and locks out your ability to pick. From a BGP perspective, each site will need to be its own ASN in the private block.

One thing I did to make sure I didn't lock myself out was add a firewall rule on each box that temporarily exposed the GUI to my own public IP and not the entire outside world. Not sure if you can do that in the production environment you're in....but worked great for our home environments. :D
 
Last edited:
Gotcha. Good point. So each site will have 3 addresses total, each paired to one of the other three sites. Ok. Scratch off that /24 and use segments from it.

edit: and while it’s a work lab, it’s partially based out of home environments :). We can forward the port temporarily to make it easier to fix. All sites have OpenVPN installed too, but I do worry about that routing going whack if we totally fat finger something.
 
Last edited:
Ok. So far this isn't working. Disabled the policy based tunnels and tried to enable VTI - configured on both sides, but they're not talking BGP across the "link".

Two VTI addresses are 172.16.254.1/2 - and they're not pinging.
 
Site D looks the same, except the two addresses on the P2 are reversed (of course).
 
Tried following hte guide more -
1620403782621.png


Still not working (That's still site A).

And now I'm moderately stumped.
 
Ok. Next update: Discarded BGP, used static routes - that made it work for the static routes defined (yay). But BGP still isn't peering across for some reason. Digging into that now.
 
You must log in or register to reply here.
Back
Top