What Diagnostic Process to Reveal MTU as Culprit on IpSEC VPN Tunnel?

[SamirD]

I'd like to know what process is typically used to discover when MTU is causing speed issues on an Ipsec VPN link. I have mtupath.exe and mturoute.exe and iperf. What should one specifically look for that points to an mtu issue when using these tools?

Thank you in advance for any replies.

 

[green91]

What is your concern? Not realizing maximum throughput across VPN tunnel?

You can use iperf (jperf gui for windows if you'd like) and test different window sizes with simultaneous streams to verify throughput.

 

[Valnar]

What is your concern?

yeah, what he said.

It also depends on the traffic, unfortunately. We just had a "new" problem at work (age old problem, actually) where logins to Active-Directory took 3 minutes across a site-to-site Cisco VPN. Even by setting all the appropriate MTU sizes, tcp-adust commands, PMTUD and stripping away the df-bits, the login problems didn't go away. Kerberos use UDP and usually ignore such restrictions. The only thing that worked was lowering the MTU on the actual domain controller.

 

[SamirD]

What is your concern? Not realizing maximum throughput across VPN tunnel?

You can use iperf (jperf gui for windows if you'd like) and test different window sizes with simultaneous streams to verify throughput.

Yes, mainly this. I have done iperf test and the like, but it seems like an arcane 'trial and error' way to test and fix the problem. I've also found various different results depending on what is set (which is what I would expect from a powerful tool). Any suggested commands? I'm currently just running

iperf -c iperf.gatech.edu -i 1

to get a baseline and then the same to the remote endpoint through the vpn to test.

yeah, what he said.

It also depends on the traffic, unfortunately. We just had a "new" problem at work (age old problem, actually) where logins to Active-Directory took 3 minutes across a site-to-site Cisco VPN. Even by setting all the appropriate MTU sizes, tcp-adust commands, PMTUD and stripping away the df-bits, the login problems didn't go away. Kerberos use UDP and usually ignore such restrictions. The only thing that worked was lowering the MTU on the actual domain controller.

So why the domain controller? If the problem was across the vpn? Or was it?

 

[Valnar]

I haven't had time to investigate completely, but I'd had this problem before over 12 years ago. Logins simply don't allow fragmentation of their packets. By setting the MTU to around 1380 on the affected PC's or servers, the problem went away. Of course, if you don't have a login problem, then it's something else - hence why I asked what are you experiencing?

 

[SamirD]

I haven't had time to investigate completely, but I'd had this problem before over 12 years ago. Logins simply don't allow fragmentation of their packets. By setting the MTU to around 1380 on the affected PC's or servers, the problem went away. Of course, if you don't have a login problem, then it's something else - hence why I asked what are you experiencing?

Thats interesting that it wasn't on the tunnel itself though. Maybe windows firewall was blocking pings keeping PMTUD from changing the mtu size on the fly.

No problem per se, but I wanted to see if there was anything that we keeping us from getting 100% of our bandwidth via our site-to-site tunnel. What's weird is that various different routers have different results on the same line with the same mtu settings.

 

[mwarps]

Whatever your VPN gateway devices are, they may not have the horsepower to encrypt and decrypt at line speed.

If you're getting line speed on non-VPN connections, this is what I'd check first.

 

[Valnar]

Thats interesting that it wasn't on the tunnel itself though. Maybe windows firewall was blocking pings keeping PMTUD from changing the mtu size on the fly.

I don't think PMTUD affects UDP packets.

 

[SamirD]

Whatever your VPN gateway devices are, they may not have the horsepower to encrypt and decrypt at line speed.

If you're getting line speed on non-VPN connections, this is what I'd check first.

This is what I was initially thinking until I ran some lab tests (static IPs on the wans connected to a switch to simulate fast Internet). The unit that was slower in production was actually faster in the lab test (over 50Mbit/s on a 100Mbit switch).

I tried some out-of-vpn tests too, but can't recall if I did any with the slower router. If the non-vpn tests were faster, what could have been wrong?

I don't think PMTUD affects UDP packets.

I believe you are right.

 

[DracoDan]

There's a super super easy way to test for fragementation. ping the destination end with a packet size equal to your MTU and set the "Do Not Fragment" bit (DNF).

In windows this would be something like:
ping -f -l 1380 8.8.8.8.

keep adjusting the packet size to find how big of a packet can be sent without fragmenting. If it's too big you will get:
Packet needs to be fragmented but DF set


With all this said, as long as ICMP isn't being blocked "Path MTU Discovery" (PMTUD) should take care of this for you. An alternative if you can't allow ICMP due to bogus security concerns (the government is well known for this stupidity) is to configure "Maximum Segment Size" (MSS) to limit the packet size before your tunnel. This is known as MSS clamping.

Sorry, just noticed PMTUD had already been mentioned, lol... okay, so ignore all that!

 

[SamirD]

There's a super super easy way to test for fragementation. ping the destination end with a packet size equal to your MTU and set the "Do Not Fragment" bit (DNF).

In windows this would be something like:
ping -f -l 1380 8.8.8.8.

keep adjusting the packet size to find how big of a packet can be sent without fragmenting. If it's too big you will get:
Packet needs to be fragmented but DF set


With all this said, as long as ICMP isn't being blocked "Path MTU Discovery" (PMTUD) should take care of this for you. An alternative if you can't allow ICMP due to bogus security concerns (the government is well known for this stupidity) is to configure "Maximum Segment Size" (MSS) to limit the packet size before your tunnel. This is known as MSS clamping.

Sorry, just noticed PMTUD had already been mentioned, lol... okay, so ignore all that!

Still good info.

 

[Valnar]

Using ping (icmp) to discover the maximum segment size unfortunately doesn't solve it. PMTUD doesn't work for UDP or ICMP, if I remember correctly. So unless you have some intelligence in your VPN appliance, like fragmentation before encryption, lowering the MTU on all devices on that network is a better fix, albeit a more intrusive and time consuming one. It also will alleviate your VPN device doing any fragmentation, thus causing slowness.

This is an age old problem with IPSec. If you can set it via a registry or policy push, forcing all the MTU's to be 1380 or 1400 should solve all VPN issues. For the few devices you can't change, like printers, having the VPN router perform PMTUD or fragmentation won't be too bad (if even needed. Not sure printer packets are that large).

 

[SamirD]

Thank you for the reply. Having access to the correct MSS and MTU would help somewhat though, right? Then one could set this lower MTU in the routers on both ends of the VPN tunnel.

 

[Mackintire]

Use iperf or jperf to test pushing UDP and TCP packets over the link.

At some point either it'll fragment or a router's buffer will fill and packets will start to drop.

Once you find the limits you can attempt to make the most of what you discover.

 

[SamirD]

Use iperf or jperf to test pushing UDP and TCP packets over the link.

At some point either it'll fragment or a router's buffer will fill and packets will start to drop.

Once you find the limits you can attempt to make the most of what you discover.

Thank you. What's some sample commands/switch options to try? I can run the standard iperf -c IP, but I highly doubt that's pushing anything.