Bottlenecks in this build?

[brutalizer]

What would be the first (or major) bottlenecks in this planned storage build? IOPS? Bandwidth? RAM? Or? What would prevent this build to scale? How far can it scale?

-Supermicro X10SAT with E3-1245 cpu
-32GB ECC RAM
-SSD disk as OS
-DELL 6GBps HBA card (like a IBM M1015 with two SFF8088 ports)
-Supermicro SC847e26 JBOD case with 45 consumer hard disks
-One single sff8088 cable from the HBA, so the JBOD. SAS expanders in the jbod are connected to each other.
-Solaris 11.1 using 4 x raidz3 (11 disks each), one hot spare.

 

[westrock2000]

What do you plan on using this for? Media serving? Home/work?

Unless you plan on spending some decent money on a network set the. ~85MB/s will be your first bottleneck over the network. So try not to worry about performance past that.

 

[brutalizer]

Forgot that. Yes, if I build that bottleneck away with 10GBit NICs and a switch (or cheap refurbished FC), what would the next bottleneck be? BTW, can you connect only two PCs to each other directly with 10GBit NICs, without going through an expensive switch?

Plan is nothing in particular, just a nerd thing. Curious.

 

[twistacatz]

The biggest bottleneck I see is your motherboard and CPU choice. I’m not sure what your design goals are but based on your JBOD I would imagine you plan on building or expanding to a large pool/pools now or in the future. With the E3 line 32GB is the limit, you should go with a different motherboard and E5 CPU to expand WAY past 32GB.

ZFS likes ram, feed the beast.

 

[brutalizer]

So you guess that 32GB RAM is too little? And the cpu is a bit weak? But how much cpu does it take to transfer data? Not too much? Or?

 

[twistacatz]

So you guess that 32GB RAM is too little? And the cpu is a bit weak? But how much cpu does it take to transfer data? Not too much? Or?

It's not that 32GB is too little its all relative to the size of your pools. The general rule of thumb is 1GB of RAM per TB. If your buying a JBOD that hold 45 hard drives I can only assume that you plan to eventually have more then 32TB's. The memory limit for all E3 CPU's is 32GB. When you move to the E5 line the CPU allows the use more memory.

CPU cycles is not really a big deal for modern hardware in regards to ZFS IMO. Any E3 or E5 CPU will do.

 

[madrebel]

actually ZFS loves CPU cycles and IPC a lot.

 

[patrickdk]

The first bottleneck you will hit, will be cpu, followed closely by sas bandwidth.

You might or might not say memory limits would be first or last, depending on what you use it for. As we have no idea if your working set will easily fit in 32gigs of ram or not.

But really, using an E3 cpu, is going seriously limit your available pcie lanes, so your likely limited already there from expanding nicely.

Ah yes, the pcie slots are screwed on that motherboard. You have nothing that doesn't share pcie lanes.
The x16 has 8 lanes and 8 shared lanes. and the 8 shared lanes also power the other two x16 slots.
And then all the x1 pcie slots are sharing a single x1 slot along with firewire and usb3.

 

[MysticRyuujin]

BTW, can you connect only two PCs to each other directly with 10GBit NICs, without going through an expensive switch?

Yes, if they are fiber one pair sends and the other receives, don't need a switch just hook em up criss crossed. If they are copper a simple crossover cable is all that is needed.

 

[twistacatz]

I'm sorry gents but any E3 or E5 dedicated to a zfs server will not bottleneck the system. I've ran it on both and I've never seen the CPU do much during normal operation and definitely not hold at 100%. Also I've seen super low end HP Micro servers saturate a gig link with crappy CPUs.

Depending on what type of disks you and how you setup the disks will determine if that motherboard with those PCI express slots can be saturated.

Like others have said we need more info to help you out anymore.

 

[patrickdk]

I have two E3 zfs servers, both get maxed out pretty often, with only 20 1tb disks on them, and with 32gigs ram.

I also have some dual cpu x5660 systems, that do fine. And a single cpu E5-2600 that also does well.

All of the above are running 4 gigabit links, and normally only max out 1 of those links.

How loaded down your cpu is allowed to be, depends on the latency you want. If your cpu is 100% busy, your going have horrible latency times.

Plus, I personally also need some extra cpu overhead to handle scrubs without impacting performance, and to handle multicore-gzip for offsite zfs sends.

All total, the E3's struggle to provide that for me.

 

[twistacatz]

I have two E3 zfs servers, both get maxed out pretty often, with only 20 1tb disks on them, and with 32gigs ram.

Maxed out during general usage or during scrubbing and compressing and uncompromising files?

My understanding was compression using ZFS contributed very little CPUoverhead? Also I believe CPU usage will come into play more depending on what type of NIC's and protocol you use to access your data. IE if you your using a onboard Realtek NIC and accessing your data via SMB you will see more of an impact then an Intel stand alone NIC and NFS.

In regards to scrubs I try to perform them over night, similar to a backup schedule when it will least impact my environment.

I could be totally wrong in in regards to all of this but this is what I gathered with my usage of ZFS and from what I've read over the past year. I am by no means an expert.

 

[twistacatz]

Also lets not poo on the E3 performance as it's not that bad. The high end E3's (1270 v3) will out perform low end E5's (2640) in most tests.

My issue with the E3 line is the ram limitations, otherwise I think it's a solid fit for a lot ZFS servers.

 

[m1abram]

The general rule of thumb is 1GB of RAM per TB.

There is no general rule of thumb that states 1GB of RAM per TB for zfs. ZFS likes RAM but not that much. Exception is if you are doing deduplication which you most likely do not want to do then you run into needed close to 1GB of RAM per TB however this still has to be evaluated in the use case.

 

[twistacatz]

There is no general rule of thumb that states 1GB of RAM per TB for zfs. ZFS likes RAM but not that much. Exception is if you are doing deduplication which you most likely do not want to do then you run into needed close to 1GB of RAM per TB however this still has to be evaluated in the use case.

I've heard that it's best practice from many sources. If you use Google you'll find many examples of this standard baseline. While there is no hard rules for ZFS this again is what most people would consider best practice.

I think your incorrect about duplication. For the duplication table to stay in RAM you need blocks used * 320 bytes * 4. Here is an example pulled from Aaron Toponce's blog:

"The rule is to assign 5GB of RAM for every 1TB of disk. This ratio comes from the fact that a deduplicated block seems to occupy about 320 bytes of storage in RAM, and your blocks could occupy anwhere between 512 bytes to 128KB, usually averaging about 64KB in size. So, the ratio sits around 1:208, which is where we come up with the "5GB RAM per 1TB disk" metric. So with a 10TB pool, we can expect to need 50GB of RAM for the DDT, or 200GB of RAM for the AR"

Again I'm no expert but I'm just sharing from my experience and from what I read.

 

[omniscence]

Yes, if they are fiber one pair sends and the other receives, don't need a switch just hook em up criss crossed. If they are copper a simple crossover cable is all that is needed.

You don't even need to pair attention to the fibers. The transceivers and connectors are build to just connect the right fiber to the right port unless you disassemble them. Or use the legacy screw or bajonet connectors. Or use a single-fibre module (which are rare and expensive).

 

[twistacatz]

You don't even need to pair attention to the fibers. The transceivers and connectors are build to just connect the right fiber to the right port unless you disassemble them. Or use the legacy screw or bajonet connectors. Or use a single-fibre module (which are rare and expensive).

I was thinking the same thing as I actually have 10Gb fiber from my ZFS box to my iMac. It's been a while since I set it up but I did not remember switching the pairs. However in the past I've had to switch pairs in a fiber patch panel when having issues making a connection between switch stacks. The two memories conflicted with each other so I didn't respond but good to know.

 

[brutalizer]

I've heard that it's best practice from many sources. If you use Google you'll find many examples of this standard baseline. While there is no hard rules for ZFS this again is what most people would consider best practice.

Again, there is no such rule of 1GB RAM for every TB of disk. If you are using deduplication such a rule exists - but not for ordinary usage. I have been active in the ZFS community since the very beginning and closely followed the ZFS mail lists by Sun, and have never seen such a rule. Never once have I seen a zfs developer mention such a rule (i have asked many questions on that list, and even Jeff Bonwick, father of ZFS, answered me sometimes). That rule is a misconception - and only valid for deduplication. For deduplication 1GB of RAM for every TB, is a lower recommendation, and you should probably use more RAM.

In fact, that rule can be FUD from competitors. In the beginning (5-7 years ago) there was lot of resistance from Linux supporters that claimed "ZFS is useless, it needs >4GB of RAM just to boot the server, you should forget about ZFS and instead use BTRFS". Back in those days 4GB was not common and quite expensive. There are Linux supporters still interested in spreading FUD about ZFS, that you need 10s of TB of RAM just to boot the system, and "ZFS eats cpu", so you need at least octo core just to boot the ZFS system - etc. In short, ZFS is shit and you can not use it unless you have a super computer, so you better use BTRFS instead. Back in the days, FreeBSD had a bug which consumed much RAM, but it should be debugged now. I think Linux has a problem with much RAM today, using ZFS? Can anyone confirm that Linux use more RAM than any other OS?

Fact is, if you have much RAM, ZFS will cache everything. On system with 8gb RAM or more, sometimes disks will not be touched at all - giving very high performance. If you have less RAM, then ZFS always needs to touch the disks, which degrades performance down to disk speed. I have myself run ZFS on a 1GB Solaris PC years ago without problems.

More RAM gives more disk cache, and more performance. Simple as that. Solaris system itself including ZFS, maybe use 300MB RAM or so. There is no requirement of 10GB RAM just to boot the Solaris system. The more ram, the more disk cache you have. It is not like, ZFS can not boot unless you have 1GB RAM for every TB disk. That is just pure FUD or ignorance.

 

[brutalizer]

The first bottleneck you will hit, will be cpu, followed closely by sas bandwidth.

You might or might not say memory limits would be first or last, depending on what you use it for. As we have no idea if your working set will easily fit in 32gigs of ram or not.

But really, using an E3 cpu, is going seriously limit your available pcie lanes, so your likely limited already there from expanding nicely.

Ah yes, the pcie slots are screwed on that motherboard. You have nothing that doesn't share pcie lanes.
The x16 has 8 lanes and 8 shared lanes. and the 8 shared lanes also power the other two x16 slots.
And then all the x1 pcie slots are sharing a single x1 slot along with firewire and usb3.

How do you know all about the x16 lane powering x8 lanes, etc etc - stuff? Do you have a link, so I can learn more on this?

Yes, pcie lanes will be a problem on E3 cpu. As of now, I have a GTX770 gaming card, and a Dell 6Gbps SAS external HBA card in another x16 slot. Both will run at x8 speed. So I should go to a E5 cpu to have more pcie lanes.

Have you noticed bottlenecks using only 32 GB RAM?

 

[twistacatz]

Again, there is no such rule of 1GB RAM for every TB of disk. If you are using deduplication such a rule exists - but not for ordinary usage. I have been active in the ZFS community since the very beginning and closely followed the ZFS mail lists by Sun, and have never seen such a rule. Never once have I seen a zfs developer mention such a rule (i have asked many questions on that list, and even Jeff Bonwick, father of ZFS, answered me sometimes). That rule is a misconception - and only valid for deduplication. For deduplication 1GB of RAM for every TB, is a lower recommendation, and you should probably use more RAM.

In fact, that rule can be FUD from competitors. In the beginning (5-7 years ago) there was lot of resistance from Linux supporters that claimed "ZFS is useless, it needs >4GB of RAM just to boot the server, you should forget about ZFS and instead use BTRFS". Back in those days 4GB was not common and quite expensive. There are Linux supporters still interested in spreading FUD about ZFS, that you need 10s of TB of RAM just to boot the system, and "ZFS eats cpu", so you need at least octo core just to boot the ZFS system - etc. In short, ZFS is shit and you can not use it unless you have a super computer, so you better use BTRFS instead. Back in the days, FreeBSD had a bug which consumed much RAM, but it should be debugged now. I think Linux has a problem with much RAM today, using ZFS? Can anyone confirm that Linux use more RAM than any other OS?

Fact is, if you have much RAM, ZFS will cache everything. On system with 8gb RAM or more, sometimes disks will not be touched at all - giving very high performance. If you have less RAM, then ZFS always needs to touch the disks, which degrades performance down to disk speed. I have myself run ZFS on a 1GB Solaris PC years ago without problems.

More RAM gives more disk cache, and more performance. Simple as that. Solaris system itself including ZFS, maybe use 300MB RAM or so. There is no requirement of 10GB RAM just to boot the Solaris system. The more ram, the more disk cache you have. It is not like, ZFS can not boot unless you have 1GB RAM for every TB disk. That is just pure FUD or ignorance.

No one said it was a requirement, I said "Best Practice". And if it's FUD you should build your server with 512MB of RAM and post the benchmarks.

 

[/dev/null]

Forgot that. Yes, if I build that bottleneck away with 10GBit NICs and a switch (or cheap refurbished FC), what would the next bottleneck be? BTW, can you connect only two PCs to each other directly with 10GBit NICs, without going through an expensive switch?

Plan is nothing in particular, just a nerd thing. Curious.

Plug as many as you want. Just make sure all the nics are on a single bridge.

 

[patrickdk]

There are 6 dual lane pcie switchs on that system

And then another pcie switch for the x1 ports

It's all right in the motherboard manual on supermicro website.

There is no way to know if 32gigs is enough or not, Sure it will work, just like the cpu will work, and everything else. But we where asked to id issues.

But the problem is, we don't know how large your working set is, will 32gigs be enough to hold it? Do you even care if it doesn't fit, and it runs slower.

Same for cpu, do you even care if the cpu is swamped, and data access is laggy or not?

If this is just a home system, sure, everything is *enough*, but again, wasn't the question asked

Depending on usage, you could hit a memory, cpu, or pcie(cause of all the pcie switches used) limit first.

The only way you will hit a limit on the disks, is if your doing pure streaming only, then cpu/memory won't be an issue, but pcie might be.

 

[brutalizer]

No one said it was a requirement, I said "Best Practice". And if it's FUD you should build your server with 512MB of RAM and post the benchmarks.

Obviously, the more ram disk cache you have, the faster it will be.

There are rumours floating around that ZFS wont even boot unless you have gobs of RAM. Well, that is not true. For best practices of 1GB per TB disk, that is not true either. I have never seen any such recommendation from the ZFS devs from Sun or Oracle. However, I have seen such recommendations for deduplication.

I am telling you there is no substance to that claim/best practice or whatever you call it. I suspect it is FUD from the Linux camp, trying to shoot down ZFS. Heck, the Linux camp even called ZFS a "rampant layering violation" implying it is badly designed, because ZFS is "one big monolithic mess". A "good" Linux design should have several separate layers, separate filesystem, volume manager, raid layer, etc.

Well, the problem with that layered approach is you can not guarantee data integrity if you have many separate layers with no communication across the whole chain. Errors might creep in when you go to another layer, but within layers everything might be fine. So, data leaving one layer might be corrupted, going to another layer. That is the big problem. ZFS on the other hand, is monolithic so manages knows everything, filesystem, raid, volumes, etc - that is the reason ZFS can guarantee data integrity, because the data never leaves ZFS control.

Funny though, BTRFS is also a rampant layering violation, going against all "good" Linux design. Why does BTRFS have a similar design to ZFS, when ZFS is "so bad"? And why is rampant layering violation only a problem when ZFS has it, but not BTRFS?
http://arstechnica.com/staff/2007/05/rampant-layering-syndrome/

So you maybe see that the Linux camp has tried to shoot down ZFS several times. Not Invented Here syndrome, you know.

 

[drescherjm]

Heck, the Linux camp even called ZFS a "rampant layering violation" implying it is badly designed, because ZFS is "one big monolithic mess". A "good" Linux design should have several separate layers, separate filesystem, volume manager, raid layer, etc.

I think reiser4 had that same distinction.. At the time it was a little frustrating but considering what happened I glad I did not move that direction with my storage ~5 years ago..

 

[twistacatz]

I understand your point and it's logical but it does not add up in regards to the hundred of threads I've read about ZFS performance. The way you describe ARC you make it seem like a bonus kind of like a L2ARC. I have a suspicion that an adequate ARC is necessary to pull decent performance when using ZFS. I believe a decent ARC is one of things that makes ZFS perform well, without it I fear you'll have awful performance with your rig.

Prove me wrong, when you build this machine use 512MB of RAM and show me the benchmarks.

 

[brutalizer]

I dont have 512MB RAM sticks, only 8GB sticks. It can be that 512MB is too small and makes Solaris swap all the time, unless you go for a minimal server installation. I think someone ripped out stuff from solaris to fit into 64MB RAM or so.

Anyway, I am saying that you dont need gobs of RAM just to boot the system, and I am also saying that if you have more RAM you get a larger disk cache which makes the system faster. But it is not a requirement to have a large disk cache. You do not need 32GB of RAM just to get minimal performance. I hope we can agree on these points.

How a ZFS based storage server works with 512MB RAM, I dont know. Maybe as good as a ext4 based storage server? It would be interesting to see ZFS benchmarks with different amounts of RAM, I agree on that. If someone has a server available to do that, I bet that thread will get lot of traffic. Or if www.servethehome.com or someone else did it.

 

[madrebel]

kind of stupid to even talk about installs with less than 16GB of ram. these days the spend on 'all' that ram is what, $100? for a single stick!

who gives a shit what zfs would or wouldn't do with 512MB of ram.

your car would perform poorly if you replaced the engine for a harley v-twin and 1/4 the displacement ... well, duh. why would i do that though?

 

[twistacatz]

I agree it's not a requirement man. Lets back on topic tho. If you do this build make sure you get that E5 man.

 

[brutalizer]

I already have this build today. I have identified another major bottleneck that no one has mentioned. I dont have enough of disks! I only have like 8-10 disks or so today.

But thanks for your help and input guys. I wanted to hear the experts discuss this build. Next time I might get a E5.

 

[m1abram]

I already have this build today. I have identified another major bottleneck that no one has mentioned. I dont have enough of disks! I only have like 8-10 disks or so today.

But thanks for your help and input guys. I wanted to hear the experts discuss this build. Next time I might get a E5.

that single hba may be your bottleneck if you fill it with 45 drives depending on the configuration of the array. I have 14 drives in a 2-way mirror setup connected to 2 8-channel HBAs and get seq read benchmarks of ~1.5GB/s. That one sff8088 is only good for 4x6gbit/s or 24gb/s which is 3GB/s.