Jumbo frames -- critical for performance?

M

Madwand

Gawd
Joined
Jul 5, 2006
Messages
755
In a diversion of a thread on "Load Balancing" here:

http://www.hardforum.com/showthread.php?t=1082381

longblock454 made certain claims;

longblock454 said:
While on the gigabit topic, don't forget about the advantages of jumbo frames.

Gigabit with standard frames of 1500 will only net you about 350Gbps, give or take in most common situations.

Gigabit with jumbo frames can usually saturate the connection!
Click to expand...

The discussion is fine, IMO, but shouldn't be in that thread. So here it is.

longblock454, in face of measurements to the contrary, how do you defend your claims? In which cases will it be important to have jumbo frame support in order to have good performance?

And then, how does a typical (home/mixed) network go about ensuring that jumbo frames support is available and viable? How do you mix non-jumbo capable routers with jumbo capable gear?

If you would like to rephrase your claims, please do so.

Let's have it out and get it straight. Maybe we'll learn something new.
 
Malk-a-mite said:
http://sd.wareonearth.com/~phil/jumbo.html

and...

http://searchnetworking.techtarget.com/searchNetworking/downloads/whitepaper_jumbomtu_sep2003.pdf

and because this happens more than once on this forum:
http://www.hardforum.com/showthread.php?t=990298
Click to expand...

Would you like to translates these into specific claims?

Do you agree with longblock454 that jumbo frames are critical to performance -- in LAN environments (not the WAN environments discussed in the linked papers)? That around 350 Mb/s cannot be exceeded without utilizing jumbo frames?

I'll leave the implementation difficulties for a bit while the importance of utilizing jumbo frames is clarified.
 
Madwand said:
Would you like to translates these into specific claims?

Do you agree with longblock454 that jumbo frames are critical to performance -- in LAN environments (not the WAN environments discussed in the linked papers)?
Click to expand...

..... not the WAN? What paper did you read?
However MTU related network issues are entirely avoidable with some careful planning such as adopting a rigorous approach to interface MTU assignment and applying a widely used MTU convention like 9000 bytes on all jumbo LAN/WAN links.


That around 350 Mb/s cannot be exceeded without utilizing jumbo frames?

I'll leave the implementation difficulties for a bit while the importance of utilizing jumbo frames is clarified.
Click to expand...

There was a nice graph in the pdf. I'll cut and paste in case you missed it.
jumbo.jpg
 
personally, i found no different in network speeds when going to standard to jumbo frames..
 
mjz_5 said:
personally, i found no different in network speeds when going to standard to jumbo frames..
Click to expand...

Did you separate your 10/100 network from your gige network?
 
Well,

This discussion could, and very possibly will, get complicated.

Directly from microsoft is part of my point when considering load.

Code: 
Jumbo Frame Support
Supporting larger Maximum Transmission Units (MTUs) and thus larger frame sizes, specifically Jumbo Frames, will reduce the network stack overhead incurred per byte. A 20% TCP throughput increase has been measured when the MTU was changed from 1514 to 9000. Also, a significant reduction of CPU utilization is obtained due to the fewer number of calls from the network stack to the network driver. 
[url]http://www.microsoft.com/whdc/device/network/NetAdapters-Drvs.mspx[/url]

The value of 350Mbps is from personal experience from an untuned system. This research was a couple years ago pulling data attempting to convience my previous employeer the importance of not using "consumer" networking hardware. This has changed slightly with some of the "better" consumer stuff. Even cheap NICs and switches are becomming pretty fast.

Plenty of jumbo frame advantages can be found here:

http://www.psc.edu/~mathis/MTU/

How about this graph displaying speed vs mtu?

http://www.psc.edu/~rreddy/networking/mtu.html

So back to the previous, origional post, how this one got started! If you are to the point that you are worried about load balancing at home, a switch and NIC that support jumbo frames could possibly be the answer to a more complicated network setup.

But, I won't forget the negitives, the single biggest is compatability! Each device on that same switch (assuming unmanaged, all ports common) must be at the same MTU unless you have an uplink port (which you can get for $100).
 
Malk-a-mite said:
There was a nice graph in the pdf. I'll cut and paste in case you missed it.
Click to expand...

I saw that graph. Since it doesn't match up with the performance I observe, I say it's wrong in general.

Do you choose to believe that graph instead of doing your own tests? Can you do no better than around 350 Mb/s without standard frames? What sort of hardware performs so poorly?
 
Madwand said:
Do you agree with longblock454 that jumbo frames are critical to performance -- in LAN environments (not the WAN environments discussed in the linked papers)? That around 350 Mb/s cannot be exceeded without utilizing jumbo frames?
Click to expand...

Remember, i said in most common situations, not an absolute.
 
Madwand said:
I saw that graph. Since it doesn't match up with the performance I observe, I say it's wrong in general.
Click to expand...

You exist as one data point.

If you can't afford a plasma TV right now, does it stand to reason that no member of the [H]forum can afford one, or that there might be something with your situation that might explain the problem?
 
My iSCSI SAN benches at 98MB/s sustained write over Cat5e(25ft) GigE without jumbo frames, connected by seperate VLAN on a Cisco Catalyst 4006 chassis & Intel Gigabit server nics on each end (forget the exact model off the top of my head). *shrug*
 
longblock454 said:
Remember, i said in most common situations, not an absolute.
Click to expand...

The problem here is that I've measured a number of NICs, and never seen anything as bad as 350 Mb/s for raw networking throughput, and most of this stuff is commonly-available consumer gear. So my read is that if you can't do better, you're talking about some really poor or out-dated situations, not any of the number of NICs I've seen.

Let's clarify this point -- is the 350 Mb/s figure about raw networking throughput, or aggregate throughput considering HD's. In the latter case, I'll agree, but only because the combination of HD + protocols + network + HD reduces performance (and increases load on the CPU), not because the network itself is necessarily under-performing.
 
da sponge said:
My iSCSI SAN benches at 98MB/s sustained write over Cat5e(25ft) GigE without jumbo frames, connected by seperate VLAN on a Cisco Catalyst 4006 chassis & Intel Gigabit server nics (forget the exact model off the top of my head). *shrug*
Click to expand...

i also get teh same speeds using NetCPS without jumboframes.. real world tests are

upload to server: 45MBs
download from server 33MBs

server windows 2003
windows xp sp2

i have no idea why uploads to the server are faster, but thats my score :(
 
Madwand said:
Let's clarify this point -- is the 350 Mb/s figure about raw networking throughput, or aggregate throughput considering HD's. In the latter case, I'll agree, but only because the combination of HD + protocols + network + HD reduces performance (and increases load on the CPU), not because the network itself is necessarily under-performing.
Click to expand...

Clarification always helps :D

"raw" networking performance is a fairly useless performance measure if you ask me, unless we were talking between switches or something!

Lets keep it between complete machines, it will be more meaningfull to 99.9% of the forum group.
 
longblock454 said:
This is close to what I normally see in average gear.
Click to expand...

These are not at all surprising, and often have nothing to do with jumbo frames, because the HD's are going to be the bottlenecks. Often, if you took two such systems and moved a drive from one machine to another, that's the sort of file transfer rates you'd get between the drives. Again, little to do with jumbo frames, much to do with HD's.

Moreover, pursuing jumbo frames in these cases would be a futile exercise. You'd do much better knowing, when applicable, that your network performance is fine, and you need to improve HD performance if you want faster transfers.

I know that inexpensive consumer NAS boxes need jumbo frames to perform even poorly, but they're straddled with poor-performing hardware in the first place. When they start getting relatively faster CPU's such as 600 MHz Celeron-M's, they start performing much better even without jumbo frames.
 
longblock454 said:
This is close to what I normally see in average gear.
Click to expand...

do you know why the uploading is faster??? both machines have good drives.. server has 7200.9 segate and the workstation has a raptor
 
longblock454 said:
The NAS box max was only 53MB/s (~400Mbps), at gigabit. Did I miss your point?
Click to expand...

Dammit, I think we're both going to get banned now. Looking closer now, that performance is still pretty bad. My intended point was this -- massive performance improvement in this case had nothing to do with jumbo frames. Hence jumbo frames are neither (necessarily) necessary, nor correlated with massive performance improvement here.

Looking closer at the results, the performance is still pathetic. What's this business about "small file" performance vs. "large file" performance? Either it can sustain load or it can't -- and large files are representative of sustained load. So in this case the example is shot, and the performance is still bad. No winner, no prize.

Note in addition two further points -- (1) they used a cheap RealTek PCI NIC -- to save $2-$10 I guess from a cheap Intel PCI NIC. I'm pretty sure that the Linux driver for this NIC is less than stellar in performance. It was when I last looked at one. Most modern MB's will have decent onboard GbE that doesn't go through the PCI bus.

(2) They put in a PCI storage controller at the same time. I'm beginning to think that they designed this box so that it wouldn't make their reams of crappy NAS box performance testing look too bad.

OK, that's nasty. But I'll bet that I could do a lot better, and I hope that most of us here would have better sense than demonstrated in that build.

longblock454 said:
Max for that raid card i bet, too bad they didn't do a 9K frame test.
Click to expand...

4 drive RAID 5 read performance only hitting ~50 MB/s? That's pathetic. But I don't think that even jumbo frames could do a lot for that build. But so what.. we're making assumptions based on our own biases about potential performance under different conditions, and neither point has been clearly demonstrated.

Let's get back to the original point then -- this was, jumping into a random build, that utilizing jumbo frames can significantly improve performance. I'll grant you this claim entirely for cheap NAS boxes, and use this to strengthen my claim -- that when you don't have such poorly performing hardware, that the critical important of jumbo frames is diminished to the point of not being necessary for good performance.

My example is a wholly consumer but not unintelligent build that can do 90 MB/s file transfers. 50-60 MB/s is also commonly achievable, sometimes even without RAID (for outer sectors), and these cases contradict the jumbo frame claim. Moreover, a key factor in performance here is the performance of the drive subsystem, which again contradicts the jumbo frame claim, as being necessary and critical for performance.

I ask again for clarification on the claim. For which systems specifically is the utilization of jumbo frames critical to performance? Where are your measurements which show this? And why is it that you have to go beyond pure networking benchmarks into drive subsystems to make a claim for jumbo frames, which are clearly at the networking layer.
 
Try this for an "idea" of the effects on frames sizes.

I don't have a personal way to test 9K frames at home, lets try this:

The jump from 1500 to 9000 (easy numbers here) is a factor of 6. Take your 100BT connection and change the MTU on both ends from 1500 to 250, same factor, then do some network testing.

Doing it at 100BT is also a good way to get rid of HD related bottlenecks.
 
I think for most people, they will hit hardware limits (such as disk i/o, pci bus, etc) before they really need more than 40-50MB/second.

I have 4 extra ide drives sitting around as well as a dual amd athlonxp (with pci-x) box I'm going to retire from the colocation sometime soon as well as a pci-x non-raid storage controller. In my main box I have a 3ware 9500sx-8LP. If any of you want, I can probably do some tests 2 or 3 months down the road and put my findings here.
 
Real-world test.

RTIX-NM-003:~# /sbin/ifconfig eth1 | grep MTU
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RTIX-NM-003:~# iperf -c rtix-ws-777
------------------------------------------------------------
Client connecting to rtix-ws-777, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.5.60.18 port 37361 connected with 10.5.60.10 port 5001
[ 3] 0.0-10.0 sec 646 MBytes 542 Mbits/sec


two gigabit machines (one fiber, one GigE), Cisco 6506 switch inbetween, 1500MTU network. 542megabit/sec.
 
Fint said:
Real-world test.

RTIX-NM-003:~# /sbin/ifconfig eth1 | grep MTU
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RTIX-NM-003:~# iperf -c rtix-ws-777
------------------------------------------------------------
Client connecting to rtix-ws-777, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.5.60.18 port 37361 connected with 10.5.60.10 port 5001
[ 3] 0.0-10.0 sec 646 MBytes 542 Mbits/sec


two gigabit machines (one fiber, one GigE), Cisco 6506 switch inbetween, 1500MTU network. 542megabit/sec.
Click to expand...


trying transfering a 2 gig file between machines, and see how fast it goes..
 
While this is not conclusive in any way, and I'd probably use 9K frames in such specialized contexts if possible (and so you might find similar examples utilizing 9K), I find this interesting:

http://www.wide.ad.jp/news/press/20050506-LSR2-e.html

Highlights of this achievement are as follows:

(1) Data transfer is achieved by 1500 byte standard Ethernet frame size and standard TCP protocol. Use of standards enable the use of technology used in this experiment can be limitless, i.e. Web accesses, file transfer, mail system and Grid computing.

(2) Servers used in the experiment were standard PC’s assembled with generally available PC parts, whose cost is about 2,500 US $.

(3) The achieved performance, 7.21 Gbps is limited by maximum bandwidth of I/O bus (PCI-X bus) and not by long-distant network or TCP technology. This means that the problem on high-latency, high-bandwidth TCP transfer is completely solved by “TCP optimization technology by inter-layer coordination” developed by the University of Tokyo using 10 Gbps Ethernet adapters with TCP offloading capabilities by Chelsio Communications.
Click to expand...

In this case, it seems that they were bottlenecked by PCI-X, and decided "what the heck, why don't we do it with standard frames and gear". Similarly, I think that when you're bottlenecked by the HD, etc., and not the CPU and poor networking hardware, you can say "why bother with jumbo frames".
 
mjz_5 said:
trying transfering a 2 gig file between machines, and see how fast it goes..
Click to expand...

iperf sends as much data as possible in 10 seconds. It takes factors like hard drive speed out of the picture, so you get a true idea of how fast the network is. Transferring files is often limited by the speed of the sender or receiver's hard drive, neither of which I give a crap about.
 
Madwand said:
16KB window is small for gigabit. Try setting it to 1 MB or something.
Click to expand...

You missed the point. The original argument was: "Gigabit with standard frames of 1500 will only net you about 350Mbps".

But I proved that gigabit with standard frames of 1500 got me over 500 megabit/sec. Maybe with some tweaking of window sizes I could go faster, but I only needed to beat 350.
 
We are all getting side tracked, this thread is on the importance of frame size.

Change your MTU to 200 on both ends, retest, and post the results if possible.

But to really evaluate the performance we need to know CPU% as well during the test.
 
I'm not willing to make all the changes neccessary to change my MTU on both my workstation and one of my servers, but iperf has an option to specify the max frame size, which should do the same thing (according to a ethereal sniff, each frame was only 254 bytes in size)

RTIX-NM-003:~# iperf -c rtix-ws-777 -M 200
------------------------------------------------------------
Client connecting to rtix-ws-777, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.5.60.18 port 47768 connected with 10.5.60.10 port 5001
[ 3] 0.0-10.0 sec 112 MBytes 94.0 Mbits/sec


I think using a larger MTU will allow higher throughput, with lower CPU usage, but you don't need jumbos to hit higher than 500megabit/sec.
 
My home 100BT connection.

Code: 
iperf -c 192.168.1.47 -M 200
WARNING: attempt to set TCP maximum segment size to 200, but got 536
------------------------------------------------------------
Client connecting to 192.168.1.47, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.45 port 36668 connected with 192.168.1.47 port 5001
[  3]  0.0-10.0 sec  46.6 MBytes  39.1 Mbits/sec
[root@localhost src]# iperf -c 192.168.1.47 
------------------------------------------------------------
Client connecting to 192.168.1.47, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.45 port 36669 connected with 192.168.1.47 port 5001
[  3]  0.0-10.0 sec    111 MBytes  92.8 Mbits/sec
[root@localhost src]# iperf -c 192.168.1.47 
------------------------------------------------------------
First test caused 100% CPU (watching gkrellm during the test), second 14%, that is only a 1/3, 9K frames would be 6 times bigger.

I say, jumbo frames are critical for performance.

Edit:

It appears I get the Warning no matter what I set the max unit size to, but it doesn't seem to be defaulting to 536 like it says. I believe my max settings are taking effect, the lower I go the slower the result and the higher the CPU, I think it is working.

So, in the first test above I believe the max was at the setting of 200, not 536 like the warning says.
 
longblock454 said:
My home 100BT connection.
..
I say, jumbo frames are critical for performance.
Click to expand...

While this is somewhat interesting, it's certainly not conclusive, esp. for real jumbo frames, and provides no information about which conditions make jumbo frames "critical". Claiming that they are "critical" without qualification is absurd, esp. in view of several direct claims in this thread alone of being "just fine" or better without using jumbo frames.

So far, only one person in this thread has said that he's tried jumbo frames, and that person's experience was contrary to the claim -- he found that it made no difference.

I've managed to get jumbo frames working between two workstations (it turned out that my switch revision didn't support jumbo frames, but I was able to make do with a direct wire connection). So I have some results coming which should be interesting.

If anyone else has additional data, that'd be better still.
 
Madwand said:
While this is somewhat interesting, it's certainly not conclusive, esp. for real jumbo frames, and provides no information about which conditions make jumbo frames "critical". Claiming that they are "critical" without qualification is absurd, esp. in view of several direct claims in this thread alone of being "just fine" or better without using jumbo frames.
Click to expand...

Critical yes, and I included all the qualification it needed. The 6x reduction in frame sized completely maxed out the CPU. Your thread title only listed "Performance" as the object and didn't specify gigabit.

Sorry, but you cannot deny the importance of frame size in that situation.

None of the other performance tests are even valid, they listed network throughput without the impact on the client machine or server.

I'm half tempted to buy some new gear so I can test along with ya!
 
longblock454 said:
None of the other performance tests are even valid, they listed network throughput without the impact on the client machine or server.
Click to expand...

:rolleyes: The question was: "can you push more than 350megabit/sec on gigabit with 1500MTU" and the answer is "yes". Granted, jumbo frames will result in less interrupts, thus lower CPU usage, but that wasn't the original question.
 
Fint said:
:rolleyes: The question was: "can you push more than 350megabit/sec on gigabit with 1500MTU" and the answer is "yes". Granted, jumbo frames will result in less interrupts, thus lower CPU usage, but that wasn't the original question.
Click to expand...

Sorry, your in the wrong thread, that was previous. This thread is simply the effect of frame size on performance, specifically jumbo frames, anything bigger than 1500 is jumbo.
 
longblock454 said:
I'm half tempted to buy some new gear so I can test along with ya!
Click to expand...

That's the spirit, but I'd suggest not buying new gear until you see some of my results ;) Then you can choose to question, re-test, buy, etc., being a little better informed.

Sorry if I didn't spell it out better earlier, but I think we're generally interested in gigabit, network and file transfer performance here, from the point of view of a home user, with a specific interest in understanding how jumbo frames impact such use cases.

If you're running a large mult-user environment professionally, then I'd say -- just go ahead and implement jumbo frames and stick to a "standard" value. It will likely give you some benefit somewhere and overall, not cost you much, and you're likely to have the necessary hardware and expertise to support it.

File transfer performance is a perfectly good application level benchmark in the home user context, and network throughput is definitely meaningful. I'm not interested in the argument for the sake of the argument. I'm interested in the argument for the sake of a usage context that's relevant to me and others here. If that doesn't interest you, then another thread branch is necessary -- I'm not going to start talking about 10/100 as if I really care about it, etc.
 
Maybe a bit of background is in order. I strongly stress the importance of frame size due to past experience. I have been a Systems Admin for 7 years now. The importance developed as the needs of my company grew from single servers that started processing data in the GB range, to SANs that now see 100+TB per day!

We started with standard "IS" machines, which were common Dells with Intel 1000Pro adaptors, that we used as upload stations. At that time ~350Gbps is about all you would get out of these stations. We had about 45 of them, mostly the same configuration, but a few were different. I am getting tired of these "other posts" claiming, YOUR WRONG, just because their system posted 500Gbps, their claim does not accurately represent my original (previous thread, hate to bring that up again!) of average hardware in common situations, that means an average! So quit posting unless you have a population of machines to brag about!

Madwand, your correct, the importance of this thread is to explore its relevance to us, I agree, lets do that. Since I can't take my work servers down or "play" with the configuration, I may just upgrade my home system. It is getting on a bit and does see a fair bit of traffic. Besides, good cards and switches are easy to come by now!
 
Madwand, any testing results yet?

I've ordered some gigabit NICs, but I still need to find a suitable switch. The ASUS RX3141 I would like to have is sold out everywhere!
 
What I've seen so far is that jumbo frames matter more for PCI NICs. I wanted to test this with an Intel Pro 1000 to be sure, and have ordered one. I hope to get it soon so that I can check it out as well.
 
You must log in or register to reply here.
Back
Top