Two pfsense on same switch limited to ISP transfer speed.

J

jadams

2[H]4U
Joined
Mar 14, 2010
Messages
4,086
So I have a gigabit switch with 3 ports in the same VLAN. Port 1 goes back to the Verizon ONT. Ports 2 and 3 goto pfsenses' WANs. Each WAN assigned a static IP from the ISP.

When I transfer data through the pfsenses I'm not getting gigabit speed as I'd expect. This is pretty odd.
 
so tracing the route goes out to the ISP's assigned gateway / beyond, then back in?
 
1 <1 ms <1 ms <1 ms pfsense.localdomain [192.168.1.3]
2 <1 ms <1 ms <1 ms static-xx-xxx-xx-xxx.cmdnnj.fios.verizon.net [xx.xxx.xxx.xxx]

Looks to me as though it never hits the gateway. the X's are the IP of the "other" pfsense's wan.
 
What method of transfer are you using? FTP, SMB, etc?

Looking at your second post, is the static IP referenced in #2 the IP of the Router or the other pfSense box?
 
etothe3ric said:
What is the hardware inside of the pfsense boxes? Do you know that they can transmit 1Gb sustained through the routers?

http://www.pfsense.org/index.php@option=com_content&task=view&id=52&Itemid=49.html

You will be hard pressed to find a Router that can do 1Gb/s in software.
Click to expand...

both boxes have gigabit nic's. I probably should have mentioned that. Even if they only had 100Mb nics I'm still only getting around 15Mb/sec transfer which is what our upload speed is capped at.

Nonetheless in other case studies on this forum I've achieved full gigabit SMB and FTP file transfers through pfsense.

Nate7311 said:
What method of transfer are you using? FTP, SMB, etc?

Looking at your second post, is the static IP referenced in #2 the IP of the Router or the other pfSense box?
Click to expand...

FTP. Yes in my 2nd post it is ip of the other pfsense box.
 
Do you have QoS configured on the pfSense boxes? If so, it will come into play and limit throughput. As far as it knows, the other pfSense box is on "The Internet" just like anything else.

Barring QoS the other thing is having Snort/IPS enabled. With my Supermicro Atom board I can't get much more than 300Mbit/s on good Intel NICs with IPS enabled. With it off I see 900Mbit/s.

Riley
 
No QoS enabled. Snort is enabled on one of the pfsense's but in my experience with pfsense I'm not being limited by a lack of hardware. I'm stuck @ 15Mb speed which just so happens to be my max upload speed.

The only reason there is even two at them moment is because I'm cutting over to one next week. So I'm prepping the new one. Tried this out of curiosity to see if it would match some of my other tests I've ran with wan throughput.

Vz has to be doing something. What or how; is what baffles me. Before I cut over next week in going to unplug the port to the Verizon ONT and reboot both boxes and the switch. See if the same thing happens.
 
Fyi one is a super micro atom running 1.2.3. The other is a virtualized 2.1 running on esxi with 2x 2.83ghz xeon cores and 2gb ram of an hp dl 360g5

I am 99.9% sure hardware is not the culprit here.

Fyi2. Interface info shows both as connecting @ 1000Mb.
 
The Verizon router is limiting your speed. Think about it. Your 2 pfSense boxes are both set with the Vz FiOS ONT as their gateway. Once traffic leaves the LAN on either side it's routed to the gateway (Vz CPE) and then it is its job for forward it on. Instead it finds the traffic is bound for another IP assigned to it and sends it back to the destination IP. It sounds to me like the FiOS router bandwidth policies extend WITHIN your assigned subnet, and not just to the rest of the internet.
 
Well, how are these WAN addresses configured?

If they aren't in the same subnet, I don't see how they could directly talk to each other.

Code: 
ifconfig -a
netstat -rn
on both boxes.

Edit: You could just add static routes on both pfsenses to the other's WAN address if these are static anyway.

Another basic question: Why even have 2 pfsense boxes?
 
Last edited:
Well, I can tell you for sure that the Atom will never hit the 1Gb mark with filtering.
https://calomel.org/network_performance.html might be of interest
Not sure if visualization is such a good idea if you want that kind of speed, also pfsense doesn't use 10 or 11-CURRENT (yet at least) which limits you to single core processing.
//Danne
 
Nate7311 said:
The Verizon router is limiting your speed. Think about it. Your 2 pfSense boxes are both set with the Vz FiOS ONT as their gateway. Once traffic leaves the LAN on either side it's routed to the gateway (Vz CPE) and then it is its job for forward it on. Instead it finds the traffic is bound for another IP assigned to it and sends it back to the destination IP. It sounds to me like the FiOS router bandwidth policies extend WITHIN your assigned subnet, and not just to the rest of the internet.
Click to expand...

This defies some of my basic understanding of networking.

My apologies again for maybe not including some info that I figured was assumed. Both wans have ips that are on the same subnet. The last octet is only one numerical value higher. The gateway shouldn't even come into play here. The tracert above even shows that it doesn't.

The ONT isn't a router. Its just the device that terminates the fiber and breaks out your data/tv/phone. Normally Verizon gives you a craptastic router, but I opted to not use it and go with pfsense instead. The ONT is a bit of a mystery. Verizon can communicate with it. They can reset it remotely, they can change your data from going out on the coax to instead coming out the cat5. My belief is that it has an IP just outside of the range of 5 statics i was given. I really can't know for sure.

Nonetheless I see no reason for two devices on the same subnet on the next switchport from each other to have their traffic between them hit their gateway. The only way to know for sure is to run a monitor session on the switch port the ONT is on and see what wireshark says.
 
Just to be sure, you're getting 15Mbps, i.e. less than 2MB/s? The switch ports are unthrottled? To test if traffic really stays local, just unplug the ONT.
 
TCM2 said:
Well, how are these WAN addresses configured?

If they aren't in the same subnet, I don't see how they could directly talk to each other.

Code: 
ifconfig -a
netstat -rn
on both boxes.

Edit: You could just add static routes on both pfsenses to the other's WAN address if these are static anyway.

Another basic question: Why even have 2 pfsense boxes?
Click to expand...

They're on the same subnet. One digit higher on the last octet.

Two pfsenses is only temporary. I'm doing some of my own research here too. First ofsense which is in production at the moment is running v1.2.3. Old school! I want to upgrade it but I need to minimize as much downtime as possible and I don't have another physical machine to swap it with. So at the moment I setup the virtualized one with brand new v2.1. I just finished configuration on it last night (importing VPN certs, setting up openvpn, configure snort, havp, setup mailscanner, etc...). Next week in going to cut over to it with hopefully only 30 seconds of downtime. Then I want to test out pfsync which is supposed to sync settings between two pfsenses. I'll update the original pfsense to 2.1 and pfsync all the settings back over to it. Hopefully that works else I'll just have some manual config to do again. Once settings are sync'd I'll again cut back over to the original (physical) pfsense. I'll keep the pfsync going to keep that virtualized one syncd in the event the first one goes down.

TL;DR reducing downtime and testing new features.
 
TCM2 said:
Just to be sure, you're getting 15Mbps, i.e. less than 2MB/s? The switch ports are unthrottled? To test if traffic really stays local, just unplug the ONT.
Click to expand...

Yep your bit to byte math is correct. I'm getting about 1.75-1.9 MB/sec.

I can unplug the ONT on the night of the cut over before I make the cut. I was going to reboot both boxes as well as the switch. My bet is that I get gigabit speeds... Or rather whatever speed the hardware can handle.

Which will just raise more questions!
 
jadams said:
So I have a gigabit switch with 3 ports in the same VLAN. Port 1 goes back to the Verizon ONT. Ports 2 and 3 goto pfsenses' WANs. Each WAN assigned a static IP from the ISP.

When I transfer data through the pfsenses I'm not getting gigabit speed as I'd expect. This is pretty odd.
Click to expand...

Don't take this the wrong way or anything, I'm simply asking but why would you be under the impression that you'd get gigabit speeds? You're limited to whatever your connection speed is ultimately. I believe, could be wrong though, that the performance also depends on the particular link aggregation [LAGG] protocol you're using; LACP, load balance, round robin.
 
Last edited:
Phantum said:
You're limited to whatever your connection speed is ultimately.
Click to expand...

Which is 1Gbps. Since an Atom probably can't route (and NAT?) that, it would be lower, but still a bit more than 15Mbps I'd guess.

Also, what link aggregation? As I see it, each box has one link to the switch.
 
Nate7311 said:
The Verizon router is limiting your speed. Think about it. Your 2 pfSense boxes are both set with the Vz FiOS ONT as their gateway. Once traffic leaves the LAN on either side it's routed to the gateway (Vz CPE) and then it is its job for forward it on. Instead it finds the traffic is bound for another IP assigned to it and sends it back to the destination IP. It sounds to me like the FiOS router bandwidth policies extend WITHIN your assigned subnet, and not just to the rest of the internet.
Click to expand...

It's the same as having two pcs on a LAN. Sure they both have the same gateway but they will arp for each other, see they are on the same subnet, and send direct. The same thing should be happening with the two routers.
 
/usr/home said:
It's the same as having two pcs on a LAN. Sure they both have the same gateway but they will arp for each other, see they are on the same subnet, and send direct. The same thing should be happening with the two routers.
Click to expand...

In theory, but his speed limitation coinciding with his upload cap makes me wonder if the Verizon CPE is playing arp games.
 
TCM2 said:
Which is 1Gbps. Since an Atom probably can't route (and NAT?) that, it would be lower, but still a bit more than 15Mbps I'd guess.

Also, what link aggregation? As I see it, each box has one link to the switch.
Click to expand...

Maybe we're talking about different things... I was under the assumption OP was referring to transferring data from LAN through WAN, which would be limited to the connection speed from ISP.
 
The way I see it, the OP is working with this scenario:

Verizon CPE-->Gigabit switch-->2 x pfSense boxes(each with public IP) and he's trying to FTP from behind 1 of the pfSense boxes to a box behind the other.
 
Unplug the ONT and see if any traffic gets passed. Will rule out if the traffic is going to the ISP or not.

If it's actually the ONT that's limiting the speed then that's kinda a crappy way of doing it for the ISP. Someone with enough tech know how would then be able to change their caps.
 
Phantum said:
Maybe we're talking about different things... I was under the assumption OP was referring to transferring data from LAN through WAN, which would be limited to the connection speed from ISP.
Click to expand...

No. Why? The WAN ports of the pfsense boxes are 1Gbps. It can only be a performance limitation since traffic isn't even hitting the ONT, but then 15Mbps would be kinda low.

I propose that before being able to post here, you ought to be able to quote TCP/IP Illustrated Vol. 1 in full. Seriously, it should be a must-read: http://users.757.org/~panzer/TCPIP-Illustrated-1/
 
Last edited:
TCM2 said:
No. Why? The WAN ports of the pfsense boxes are 1Gbps. It can only be a performance limitation since traffic isn't even hitting the ONT, but then 15Mbps would be kinda low.

I propose that before being able to post here, you ought to be able to quote TCP/IP Illustrated Vol. 1 in full. Seriously, it should be a must-read: http://users.757.org/~panzer/TCPIP-Illustrated-1/
Click to expand...

Right, I thought OP was talking about transferring data out to the internet which would be limited to the connection speed from ISP. I realize now that's not what we're talking about. My theory is that regardless of how it's setup that data is in fact going out through the ONT to CO and coming right back. Meaning the bandwidth available is indeed connection limited.
 
http://i.imgur.com/8alGQ9J.png

As promised. Screenshots to prove my sanity.

Unfortunately I wont be able to cut over this week as I thought I would be able to which will keep me from unplugging the ONT and doing another test. Turns out I'm on call for customer stuff this week. Thats all i need is to get a call while in the middle of cutting over our own router. Its going to have to wait till next week. I know all of you are oh so on the edge of your seats for this ;)
 
Do you really have a /24 from your ISP? Also good job obscuring the addresses. :)

At this point it's a pretty obscure problem. Is pfsense doing something like auto-tuning to provide upstream QoS? Are you _sure_ there's no QoS active?
 
TCM2 said:
Do you really have a /24 from your ISP? Also good job obscuring the addresses. :)

At this point it's a pretty obscure problem. Is pfsense doing something like auto-tuning to provide upstream QoS? Are you _sure_ there's no QoS active?
Click to expand...

Just because the subnet is /24 doesn't mean he was given that block. He said earlier in the thread he was given 5 statics. I'm on a /24 but was only give 2 statics.

Also if you are going to hide your IPs, at least hide the gateways as well ;) Usually they are in the same subnet.
 
I'm a FiOS and pfSense expert, and I can't even explain this one... Maybe try another switch?

You're clearly looping through the PON for this "local" traffic... But why is super strange, I have never seen this happen like this considering pfsense is seeing everything as being perfectly configured.

You could add more trunked VLANs to both pfsense instances and static route them to each-other...
 
/usr/home said:
Just because the subnet is /24 doesn't mean he was given that block. He said earlier in the thread he was given 5 statics. I'm on a /24 but was only give 2 statics.
Click to expand...

That's the problem. If the subnet is configured as /24, he won't be able to reach any hosts "in" this /24 that are not his own. You shouldn't configure broader netmasks than what is routed to you.

Edit: Think about it. If an interface is configured with a /24, you're telling the kernel via the routing table that all hosts in this subnet are directly reachable and need to be looked up using ARP - which fails since those hosts are not on your segment. That's the whole point of subnet masks. You don't blindly configure everything with /24. It actually is done for a reason. As I said, TCP/IP Illustrated, read it.
 
Last edited:
Is there any possible way that the pfsense box(es) somehow have the default gateway as a higher priority then regular subnet traffic? Or perhaps, that somehow same subnet traffic isn't even ON the routing table? I guess you would have get to the terminal on the pfsense to check this, im not sure which commands though.
 
serpretetsky said:
Is there any possible way that the pfsense box(es) somehow have the default gateway as a higher priority then regular subnet traffic? Or perhaps, that somehow same subnet traffic isn't even ON the routing table? I guess you would have get to the terminal on the pfsense to check this, im not sure which commands though.
Click to expand...

Not in any sane kernel I know. Since you don't even know the commands (netstat -r), maybe leave the wild speculations out.
 
TCM2 said:
Not in any sane kernel I know. Since you don't even know the commands (netstat -r), maybe leave the wild speculations out.
Click to expand...
Are you saying it's actually impossible to be setup that way even if you tried, or that any default setup would never have that?
 
The default route is the most generic route. Basically any other route will take precedence over it.

If you configure an address on an interface, you get a routing table entry that overrides the default route:

Code: 
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
192.168.1/24       link#10            UC         5        0     -     4 vlan150
192.168.1.1        00:0c:29:xx:xx:xx  UHLc       0   565337     -     4 vlan150
192.168.1.254      00:0c:29:xx:xx:xx  UHLc       0        4     -     4 lo0

Any host in this subnet is discovered using ARP and an entry for it is placed into the routing table as well, further overriding the subnet route.

This is network 101 and all explained in detail in the book - which is why I'm compelled to refer to it for the 3rd time now in a sub-forum called "Networking", which is kinda sad.
 
TCM2 said:
The default route is the most generic route. Basically any other route will take precedence over it.

If you configure an address on an interface, you get a routing table entry that overrides the default route:

Code: 
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
192.168.1/24       link#10            UC         5        0     -     4 vlan150
192.168.1.1        00:0c:29:xx:xx:xx  UHLc       0   565337     -     4 vlan150
192.168.1.254      00:0c:29:xx:xx:xx  UHLc       0        4     -     4 lo0

Any host in this subnet is discovered using ARP and an entry for it is placed into the routing table as well, further overriding the subnet route.

This is network 101 and all explained in detail in the book - which is why I'm compelled to refer to it for the 3rd time now in a sub-forum called "Networking", which is kinda sad.
Click to expand...
thx. I'm sorry not all of us are up to snuff on networks as you are, many of us are constantly learning still (im sure you understand). I, unfortunately, do not have the time to read that in the immediate future, maybe later. But I still try to offer possible suggestions when I see an unsolved problem. Part of the learning process and all.
 
TCM2 said:
That's the problem. If the subnet is configured as /24, he won't be able to reach any hosts "in" this /24 that are not his own. You shouldn't configure broader netmasks than what is routed to you.

Edit: Think about it. If an interface is configured with a /24, you're telling the kernel via the routing table that all hosts in this subnet are directly reachable and need to be looked up using ARP - which fails since those hosts are not on your segment. That's the whole point of subnet masks. You don't blindly configure everything with /24. It actually is done for a reason. As I said, TCP/IP Illustrated, read it.
Click to expand...

Or the ONT is responding to ARP requests forcing traffic through it. Why? Who knows.

If this scenario will be a common one, why not slap another NIC in both boxes and route around the ONT?
 
You must log in or register to reply here.
Back
Top