IFConfig WAN Port RX Errors & Overruns when using WireGuard - should I be concerned?

[OpenSource Ghost]

Ubiquti UniFi Dream machine here (with latest stable firmware). IFConfig reports RX packet errors and overruns for WAN port, but only when using NordVPN WireGuard and when download speeds are about 200Mbps or above. NordVPN WireGuard is used exclusively on client devices (Windows, Android, iOS), not the router itself.

IFConfig never reports any errors and/or overruns when WireGuard VPN isn't used, even at the highest download speeds (500Mbps) my ISP allows. Normally having RX errors and overruns means (based on Google search) is that the router is receiving traffic at a rate higher than it can process it, but in this case only when WireGuard VPN is used. Disabling IDS/IPS makes no difference.

Windows OS doesn't register any packet loss within the VPN tunnel itself. Latency in games is also great (almost identical to latency without VPN). Maximum speed with WireGuard VPN is about 90% of the highest speed my ISP allows.

Should I be concerned? I assume that the router simply can't process WireGuard packets fast-enough because they are encrypted. Should I ask more on Ubiquiti forums or should I contact NordVPN?

 

[toast0]

Overruns definitely sounds like CPU exhaustion. I'm not familiar with Ubiquiti hardware/software to know if there's anything you can do about it though. Reducing network processing, if possible, would help.

I checked the datasheet, and the device is quad core, but since Wireguard puts everything from client to server over one UDP connection, that's likely all going to be pinned to a single CPU, which could be why your maximum speed is lower than when you connect directly, and connections tend to be more widely distributed among the CPUs.

But you should probably check with ubiquiti forums and see if there's any configuration advice to help with this case.

 

[OpenSource Ghost]

MTU is the issue. If I use MTU of 1500 (default for my ISP) in Windows, then router reports a ton of WAN port RX errors and overruns with NordVPN's WireGuard VPN, WireShark reports a ton of fragmentation, but Windows itself does not report any fragmentation (netstat -s). If I set Windows MTU to 1420 (standard for WireGuard protocol), then I achieve high throughput with WireGuard much faster, router reports no RX overruns, errors, WireShark also reports no fragmentation, but Windows (netstat -s) reports fragmentation without packet loss.

If I set WAN port MTU to 1420 (via SSH ifconfig command), then I get disconnected from VPN regardless of MTU set in Windows.

How can WireShark detect fragmentation when Windows doesn't and not detect fragmentation when Windows does? I think fragmentation can be a per-interface aspect because WireGuard uses TUN virtual adapter interface. Is there some way to see fragmentation on per-interface basis with a netstat command? Netstat -s only shows overall summary.

 

[toast0]

Ok this is starting to make sense. Wireguard is a UDP tunnel, and if your tunnel MTU is set too high, when you send or receive full MTU packets in the tunnel, they'll be too big to send as a single packet after wireguard encoding; so then they'll be sent as IP fragmented UDP. The firewall needs to defragment before it can process the packets, and then it may or may not refragment to send on the LAN.

All that defrag/refrag stuff is going to eat your cpu, and you'll get overruns and what not.

Your maximum tunnel MTU should be set to the actual path MTU between you and the VPN endpoint - the bytes of overhead. 1420 is probably under that for most networks.

 

[OpenSource Ghost]

Actually, I was wrong about MTU and you were right about the singular connection. Changing MTU only worked briefly. I inquired from my ISP and their tech stated that "Its the way quality of service demand load balancing works". They advised me to first run some speed tests without VPN until bandwidth gets bumped to 100% of ISP-advertised speed and then test my VPN connection.

It works. Every time I leave my network idle for a while and run speed test within VPN tunnel, download speed stays at 50% of ISP-advertised speed no matter how many speed tests I run. It just doesn't change, even if I run speed tests for a whole hour via the same VPN connection. If I follow tech's advise to run speed tests without VPN first, then my first speed test result is almost always 50% of ISP-advertised speed, but my second and third reach 100% of ISP-advertised speed. After second/third test (that reaches 100% of ISP-advertised speed), I re-launch my VPN and voila and get full speed with it!

Errors and overruns count still goes up (when VPN is used), even when my speed is at 100% of ISP-advertised. Ratio of "RX packet overruns / RX packetrs transmitted" is about 0.003% when I get half of ISP-advertised speed in VPN tunnel, and about 0.0001% when I get full ISP-advertised speed. There is never a single overrun or error when running full ISP-advertised speed outside of VPN tunnel.

I get the same half-speed results within VPN tunnel if I set ISP gateway to router mode and connect to it directly. Once again, if I get out of VPN tunnel and run speed tests until ISP-advertised bandwidth is achieved (1-3 tests), then re-connecting to VPN results in full ISP-bandwidth within the tunnel, minimal errors and overruns. Therefore, it is unknown whether the "QoS demand load balancing" is the issue on my router's end or on ISP's end, but a singular VPN conneciton isn't able to raise demand for full ISP-advertised bandwidth.

Having to "rev up VPN connection speed" with multiple out-of-VPN-tunnel speed tests is a ridiculous concept, even if "revving up" only takes 10-20 seconds...

 

[Nobu]

Actually, I was wrong about MTU and you were right about the singular connection. Changing MTU only worked briefly. I inquired from my ISP and their tech stated that "Its the way quality of service demand load balancing works". They advised me to first run some speed tests without VPN until bandwidth gets bumped to 100% of ISP-advertised speed and then test my VPN connection.

It works. Every time I leave my network idle for a while and run speed test within VPN tunnel, download speed stays at 50% of ISP-advertised speed no matter how many speed tests I run. It just doesn't change, even if I run speed tests for a whole hour via the same VPN connection. If I follow tech's advise to run speed tests without VPN first, then my first speed test result is almost always 50% of ISP-advertised speed, but my second and third reach 100% of ISP-advertised speed. After second/third test (that reaches 100% of ISP-advertised speed), I re-launch my VPN and voila and get full speed with it!

Errors and overruns count still goes up (when VPN is used), even when my speed is at 100% of ISP-advertised. Ratio of "RX packet overruns / RX packetrs transmitted" is about 0.003% when I get half of ISP-advertised speed in VPN tunnel, and about 0.0001% when I get full ISP-advertised speed. There is never a single overrun or error when running full ISP-advertised speed outside of VPN tunnel.

I get the same half-speed results within VPN tunnel if I set ISP gateway to router mode and connect to it directly. Once again, if I get out of VPN tunnel and run speed tests until ISP-advertised bandwidth is achieved (1-3 tests), then re-connecting to VPN results in full ISP-bandwidth within the tunnel, minimal errors and overruns. Therefore, it is unknown whether the "QoS demand load balancing" is the issue on my router's end or on ISP's end, but a singular VPN conneciton isn't able to raise demand for full ISP-advertised bandwidth.

Having to "rev up VPN connection speed" with multiple out-of-VPN-tunnel speed tests is a ridiculous concept, even if "revving up" only takes 10-20 seconds...

Well they get a lot of VPN traffic that likes to bounce around between hosts before reaching its destination, a fair amount of which is people watching videos or downloading large files. If that same traffic was allowed to take a more optimum route, it could reduce the load on the network considerably (depending on exactly how much, and whether they hit the same nodes). So I understand why they throttle sustained high-bandwidth traffic that's in a VPN, even if I don't necessarily like it.

 

[OpenSource Ghost]

Overruns and errors during speed tests within VPN tunnel only happen initially. I think such an activity is called a micro-burst. Some sources say it isn't uncommon to produce overruns and errors for whichever interface during such bursts.

Gradual increase in speed (within VPN tunnel) to ISP-advertised speed doesn't produce any overruns and errors, but I can't find a service that can run a speed test for a prolonged time (5 minutes+) to test this correctly. I test by downloading large files and see no overruns and/or errors when maximum bandwidth is achieved over time.