Fast NAS 100+ MB/Sec Design Help

[dreadwing]

Hi,

I currently have storage segmented across multiple devices and am looking for a shared storage solution.

Does anyone have a good design for a homebuilt NAS device? I would like fast file transfers with redundant storage. A list of equipment and software to use would be very helpful. Here is a list of requirements. Budget is $1500 or less.

- Fast speeds of 100 MB/Sec read write if possible
- Shared storage for ESX 4.1 and Hyper-V
- Movie media storage
- Quiet and cool operation
- 10-15TB of redundant storage that can be expanded later to 20 TB+
- What raid type & Sata Raid Card should I use, as well as motherboard, case, cpu etc...



Thanks in advance,

DW

 

[danswartz]

Specific HW is a wide area. For SW I would recommend ZFS: OpenIndiana + napp-it. Depending on how many drives you need, one or more IBM M1015 cards in JBOD mode (or flashed to IT firmware.)

 

[leeleatherwood]

With my hardware I get 110MB/sec Reads using SMB/CIFS and 80MB/sec Writes, I think my Realtek NIC in my desktop that is limiting my write speed.

I am using the motherboard onboard LAN, its a Broadcom chip and its fast. At first i was using an Intel 1000MT but couldnt get over 80MB/sec on reads or writes, tried the onboard ethernet and it works great.

http://www.newegg.com/Product/Product.aspx?Item=N82E16813186168 (i got it for $49)

http://www.newegg.com/Product/Product.aspx?Item=N82E16819103903 (its actually way overpowered, I never use above 50% usage even when writing at 500MB/sec)

http://www.newegg.com/Product/Product.aspx?Item=N82E16817182131 (modular and has 8 SATA connectors)

http://www.newegg.com/Product/Product.aspx?Item=N82E16820231426 (i got 2 of these for $34 each, but 8gb should be plently)

http://www.newegg.com/Product/Product.aspx?Item=N82E16822145475 (8 of these for 16TB total)

http://www.newegg.com/Product/Product.aspx?Item=N82E16811146071 (very nice case for a NAS)

http://www.servethehome.com/intel-sasuc8i-lsi-lsi-sas3081er-lsi-1068e-based-raid-controller-review/ ($60 on Ebay, flash it with the IT-Firmware)

Software is FreeNAS. I do like napp-it but I just couldnt get the same performance as FreeNAS gives me.

Quick transfer of a DVD to and from my NAS, as you can see the reads hover basically at 110MB/s sec and hold steady. The writes at 80MB/sec but like I said, I think my Realtek NIC in my desktop is the cause of that.
That first red spike you see is actually a 500MB CD Image, as you can see it was very fast, I had to find a larger file to test with.

ZFS Write a 20GB file, +500MB/sec

[admin@GENESIS] /# dd if=/dev/zero of=/mnt/RaidZ/ddtest.test bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes transferred in 40.419369 secs (518848278 bytes/sec)

ZFS Read a 20GB File, +700MB/sec

[admin@GENESIS] /# dd of=/dev/zero if=/mnt/RaidZ/ddtest.test bs=1M count=20000
20000+0 records in
20000+0 records out
20971520000 bytes transferred in 29.182940 secs (718622592 bytes/sec)

 

[Godmachine]

I can not find the Intel SASUC8I for less than $153 on Ebay. What's another cheap RAID card that's as good?

 

[leeleatherwood]

I can not find the Intel SASUC8I for less than $153 on Ebay. What's another cheap RAID card that's as good?

The Intel SASUC8I is just a rebranded LSI 1068e based card. The IBM br10i is practically the same as the LSI 1068e with a few very minor differences. I am using the br10i flashed with IT-Mode firmware. Basically it disabled the onboard RAID and turns the card into a HBA (host bus adapter) which is exactly what you want for ZFS.

http://www.servethehome.com/ibm-serveraid-br10i-lsi-sas3082e-r-pciexpress-sas-raid-controller/

Here is a br10i for $50
http://www.ebay.com/itm/IBM-44E8689...332?pt=LH_DefaultDomain_0&hash=item27be6eb2dc

the IBM M1015 like danswartz said is good too, the M1015 is a rebadged LSI 9240-8i card. It is newer and faster (SATA3) and costs slightly more. Nothing wrong with a M1015 but since mechanical hard drives cant even max out SATA1 you might as well save a little monies and go with the 1068e cards.

EDIT: the IBM1015 is supposed to have better support for SAS expanders though, so if you ever feel in you future you possible could use a SAS Expander, grab the M1015.

 

[davros123]

definitely doable.

my setup is as follows...
E3-1230
Supermicro X9SCM-F Mobo, 4x pcie, C204, IPMI and iKVM
16GB ECC Kingston KVR1333D3E9S/4G
Intel dual CT nic
IBM M1015 – flash with P10 firmware (~100 on ebay).
Crucial M4 128GB for the VM's
esxi on a 1 GB usb (already have an slc stick)
solaris 11 express as a san (perhaps..)
passthrough of 10 3TB drives to Opensolaris san
Main VM's running off the SSD...other, less critical/play vm's running off san via nfs shares.
other vm's running off san via iscsi
Will be running 4 linux, 2 windows vm's full time and others for playing/testing.

 

[hotzen]

@davros123, this is quite the setup i am planning, a few questions:
which chassis do you use?
why the additional Intel CT? Regarding dual, really a CT or a PT?

 

[s0rce]

Here is a M1015 for $85: http://www.glcomp.com/ibm-system-x-serveraid-m1015-sas-sata-controller

My NAS (which gets 100mb+ via SMB/NFS) is:

CPU: AMD Athlon II X2 250
HSF: AMD OEM cooler from an A8-3850
MB: Asus M4A785T-M
RAM: 4x Super Talent DDR3-1333 4GB ECC
SAS Controller: IBM Br10i
NIC: Intel CT gigabit Ethernet
RPool: Mirrored 16GB Kingston SSD
RaidZ: 4x Hitachi 2TB 5k3000 + OCZ Vertex 2 64gb L2ARC
Case: Coolermaster Elite 343
PSU: SFX Seasonic 350W
OS: Solaris 11 Express

 

[dreadwing]

Thanks for the info. Can you modify the recs for a dedicated NAS for low power, heat, and noise? Based on the replies here is what I've come up with but please adjust as needed:

- Micro atx board (suggestions plz)
- i3 cpu for low heat/power or Celeron / dual core (suggestions?)
- Lian Li's PC-A17B case w/ icy 5-in-3 drive cages
- 8 x HITACHI Deskstar 5K3000 Raid 5
- 8GB RAM
- Intel EXPI9402PT PRO/1000 PT Dual Nic Card (Pci express) for link aggregation
- IBM M1015 or dell perc6i sata card which I currently have an extra of (will this work?)
- PSU - 650W Modular (is this overkill)

Software:

FreeNas with using LACP (where do you install, on HD or USB stick?)

Is it best practices to carve out the full storage using SATA card bios then split it up using FreeNas? I would like carve out some storage for VM's and the rest for Media and personal Data. How do most of you handle dividing out storage?

Thank for the replies, great info.

 

[ChrisBenn]

E3-1220L is a 20W Xeon - and you can still get ECC ram with that. Only problem is finding a source for the process.

Otherwise i3-2100T (T is the low power/heat flag). http://www.newegg.com/Product/Produ...le-_-Processors+-+Desktops-_-Intel-_-19116394

Motherboard - http://www.supermicro.com/products/motherboard/Xeon/C202_C204/X9SCL_-F.cfm
also comes with dual nics so no need for the extra Intel nic card (and this board comes with two 82574L nics, which are both supported in OI/FreeBSD - some of the newer intel nics aren't supported yet)

If you are going to run the VM's on the server I would just go ahead and max out the memory at 16GB - otherwise 8 is fine.

Re: perc 6i - I think you will have to create 8 Raid-0 luns with that card, one for each disk. It will work, but is generally considered non-optimal. With the M1015's being affordable and available that would be my suggestion.

Divide out storage using zfs, you really want minimal bios interaction (just pass through the disks).

Re booting: I boot of a set of mirrored 2.5" drives. I had performance issues booting off of USB.

 

[dreadwing]

So the i3-2100T will work with the supermicro X9SCL+-F mobo, correct? Looks like that e3-1220L is only in the UK. Whats the benefit of the 1220l over the i3-2100T?

I am thinking about just running either FreeNas or OpenSolaris, so would the above hardware be overkill?

Based on the feedback, I will skip the perc6 and use the m1915. So then use JBOD and then use RaidZ/ZFS to handle the raid/storage config.

For OS, cheap ssd's mirrored should do the trick?

 

[Jonowxeno]

So the i3-2100T will work with the supermicro X9SCL+-F mobo, correct? Looks like that e3-1220L is only in the UK. Whats the benefit of the 1220l over the i3-2100T?

I am thinking about just running either FreeNas or OpenSolaris, so would the above hardware be overkill?

Based on the feedback, I will skip the perc6 and use the m1915. So then use JBOD and then use RaidZ/ZFS to handle the raid/storage config.

For OS, cheap ssd's mirrored should do the trick?

the i3 2100t will work in the X9SCL-F. Benefit of the e3-1220l(or any xeon) is ECC RAM

The 650W is overkill for just 8 drives, but i find looking for anything smaller tends not to be modular and/or cost more (looking at gold rated)

If this is just for home media sharing, I wouldn't bother mirroring two SSDs for OS, just get 1 and make a offline backup or snapshot if doing VMs

 

[dreadwing]

If I can get a deal on the e3-1220l i woudl go with that. If I pick the 2100t, i can use the x9scl-f board but would have to use non-ecc ram, correct?

Thanks for the tips.

 

[ChrisBenn]

If I can get a deal on the e3-1220l i woudl go with that. If I pick the 2100t, i can use the x9scl-f board but would have to use non-ecc ram, correct?
Thanks for the tips.

Yep - board will work with either. I think actually you need ECC ram with that board regardless of if the CPU supports it:
ref:
http://www.webhostingtalk.com/showthread.php?t=1071103

If you can't find a 1220l there's also the 1260l
http://www.provantage.com/intel-cm8062301061800~7ITEP3J6.htm
it's available - ~300, so about 2x the price of the i3-2100.

Edit: Actually I wonder if the 2100t will support ECC with a server chipset (like the supermicro board) - on ARC the ECC slot doesn't say yes/no, but is just blank.
http://ark.intel.com/compare/53401,53423

it is 4 core/8 thread (vs 2/4 for the i3-2100t). If you plan on using the box as a VM host it would be worhwile - otherwise probably overkill but it's only 40w (i3-2100t is 35w)



I wouldn't bother with SSD's for the OS drive - just bog standard 2.5's are fine - 2 http://www.serversupply.com/products/part_search/pid_lookup.asp?pid=120929 for instance (Still less than 1 SSD) - 30 bucks each. I would tend to go with 2 mirrored drives for boot - not because you can't backup/recover your boot partition, but just to save you the PITA factor if it goes down.

If you want to do anything with SSD's you could add one as an L2ARC drive.

 

[dreadwing]

I think if I go with the i3-2100T, I would need to go with a non-ecc board. Any reason to use the X9SCL-F aside from ecc memory and the internal dual lan?

The mirrored boot drive is probably a good idea and I plan to use a SSD for L2ARC. Will test some option on my current vm's to see what solution I like best.

If I do passthough JBOD and use either raidz or zfs, I can always add more drives to the array, right? I plan to start with 6 of the Deskstar 5K3000 and expand from there.

 

[Dangman]

I think if I go with the i3-2100T, I would need to go with a non-ecc board. Any reason to use the X9SCL-F aside from ecc memory and the internal dual lan?

Well the NIC used by the X9SCL-F is Intel which means it's one of the fastest NICs out there which will further help you reach your goal of fast LAN transfers. Remember that a standalone Intel NIC is about $30. It also has builtin IPMI 2.0 with virtual media over LAN and KVM-over-LAN support (you can access the BIOS and install the OS from a remote computer, no keyboard, mouse or monitor needed).

If I do passthough JBOD and use either raidz or zfs, I can always add more drives to the array, right? I plan to start with 6 of the Deskstar 5K3000 and expand from there.

To the ZFS/RADIZ array, no. To the ZFS/RADIZ pool, yes. This quote explains it:

What you cannot do, is expand an existing RAID-Z (RAID5) or RAID-Z2 (RAID6) array with one or more disks.
But, you can add new disks or RAIDs to an existing pool. So if you have a 4-disk RAID-Z, you can add another 4-disk RAID-Z so you have 8 disks. The second array would share free space with the first; in essence it would be a RAID0 of two RAID5 arrays. ZFS can expand this way.

What you can do, is expanding mirrors and RAID0's. In the example above that's what actually happened: a new array is RAID0-ed with the existing array. New created files will be written to both devices, for additional speed. Setting copies=2 would make files in that directory be stored on both RAID arrays; for extra redundancy.

- Lian Li's PC-A17B case w/ icy 5-in-3 drive cages

Exactly how many cages are you planning on getting?

 

[dreadwing]

For the X9SCL-F trying to confirm it can support the i3-2100T, the PDF for the mobo says "2nd generation Intel Core i3.

Kinda confused about if you can expand the volume by additing more disks, from your response, it sounds like u need to make an addional array that you can share space with an existing volume.

I plan to start with 2 icy cages or get a case that has 8 X 3.5 drive bays and forgo the icy bays until needed.

Still in design phase, I already have 7x 2TB Deskstar 5K3000, still need sata card, case, mobo, cpu, and ram. Fun Stuff.

 

[Dangman]

Kinda confused about if you can expand the volume by additing more disks, from your response, it sounds like u need to make an addional array that you can share space with an existing volume.

That's basically it.

I plan to start with 2 icy cages or get a case that has 8 X 3.5 drive bays and forgo the icy bays until needed.

Well here's the deal: If those Icy cages are at all part of your plan, they're not cost-effective. Right now you can get the following case for $330 shipped:
$330 - NORCO RPC-4220 4U Rackmount Server Case

The above case allows for 20 hard drives. Once you remove the side handles, you can easily stand up the case like a normal ATX case. So keep in mind that if you want hot-swap capability with the most bang for the buck, the Norco is it.

However, if you forgo hot-swap capability and don't plan on expanding past 12 hard drives, I recommend the following setup:
$50 - NZXT Source 210 Elite Black ATX Case
$24 - Cooler Master STB-3T4-E3-GP 4 in 3 Device Module w/ 120mm Fan
---
Total: $74 shipped.


PSU wise, I recommend this:
$106 - SeaSonic M12II 620 Bronze 620W Modular PSU

 

[davros123]

@davros123, this is quite the setup i am planning, a few questions:
which chassis do you use?
why the additional Intel CT? Regarding dual, really a CT or a PT?

hi hotzen, I am using a LianLi A77 care, it's a huge case with loads of room...but I do not think they make it any more...most seem to go for a norco if they need loads of drives.

as for the nic, yes, it's a PT, my bad...., I do not use it at this stage...as the mobo. has two nic's and you can tweak the esxi install iso to recognise both nics. I will use it later as I plan to virtualise my firewall too.

cheers.

 

[Godmachine]

Thanks for the info

 

[Pitbull]

You do not need, 16 or even 8 of memory for NAS, i have a nas setup at work that also hosts few other things on an older dual core dual xeon that will max out multiple gbit links easily, 7.2k rpm drives can do 70+ MB each so having 8 of em in raid will be more then enough, just dont use cheapo on board gbit nics.

 

[Trepidati0n]

You do not need, 16 or even 8 of memory for NAS, i have a nas setup at work that also hosts few other things on an older dual core dual xeon that will max out multiple gbit links easily, 7.2k rpm drives can do 70+ MB each so having 8 of em in raid will be more then enough, just dont use cheapo on board gbit nics.

I agree that you don't need 8GB+ of memory but ZFS based system do benefit from it.

 

[ChrisBenn]

You do not need, 16 or even 8 of memory for NAS, i have a nas setup at work that also hosts few other things on an older dual core dual xeon that will max out multiple gbit links easily, 7.2k rpm drives can do 70+ MB each so having 8 of em in raid will be more then enough, just dont use cheapo on board gbit nics.

Need is relative - you don't need 100MB/sec, 70MB/sec, heck 30. But it can improve the experience.

If all you are doing is bulk sequential transfer of large files (copy a movie to my hard drive, stream it off) then yeah. But say for instance you have something that is going to generate lots of small random requests - say you have a lightroom picture directory on the NAS, or you are generating thumbnails, or your itunes library is reading metadata. In those cases it's not so much about your bulk transfer rate. And in those cases having more memory *can* help. ZFS will use the extra memory as an "ARC" cache - hueristically determining which blocks to store in there - and if/when you request those blocks they don't have to be read off of the disk, but can be returned from memory.

If you start using more advanced features system memory can be required for them as well - specifically L2ARC and deduplication.

So is the memory needed? No - is it useful and will it provide a performance benefit - probably.

And really, memory is so dirt cheap now why would you not throw at least 8GB in there (assuming you aren't on an atom)

 

[Pitbull]

I'd rather have 4g of ECC/registered memory on a nas server then 8g of junk, reliability comes to mind, i do not want the NAS to drop dead because of cheapo HTPC based hardware. But i guess for home use that's fine.

 

[ChrisBenn]

I'd rather have 4g of ECC/registered memory on a nas server then 8g of junk, reliability comes to mind, i do not want the NAS to drop dead because of cheapo HTPC based hardware. But i guess for home use that's fine.

Huh? When did this become about ECC vs. non ECC memory? I thought it was about the advantages & utility of more memory. I would expect whether you run ECC or not is really going to be dictated by your build (did you buy a processor/mobo which support it). If you have support for it then you would have to be crazy not to run it.

But that's a completely different discussion - the point was that ZFS systems in non purely sequential workloads can benefit from more memory.

 

[Trepidati0n]

Huh? When did this become about ECC vs. non ECC memory? I thought it was about the advantages & utility of more memory. I would expect whether you run ECC or not is really going to be dictated by your build (did you buy a processor/mobo which support it). If you have support for it then you would have to be crazy not to run it.

But that's a completely different discussion - the point was that ZFS systems in non purely sequential workloads can benefit from more memory.

It became that when you called him on his overreaching memory argument. Instead of saying "you have a point" he tries to change the argument.

 

[PointandClick]

100MB/s is easy for the NAS box itself. You're going to be limited by your network and your client machines.

My box consists of an Athlon II X4 635, ASUS M4A79XTD EVO, 4gb ecc ram, and 3 Samsung HD204UI. The X4 is overkill but it was only a few bucks more in a combo deal. With only 3 drives in Raidz I get about 170MB/s on the host box for both reads and writes. Using dd from my desktop in OSX I get about 100MB/s write and slightly lower reads. This is with realtek nics on both boxes

 

[dreadwing]

Is using a zraid/zfs comapred to a traditional raid 5 still the preferred method? I read somewhere that an issue with zraid is that the IOPS performance is only the amount of a single drive and not the whole array, is that correct. So, does L2ARC help improve IOPS?

Aside from using the NAS for media and movie streaming, I woud like to host ESX vms. Am I better off with a raid 5? I am still on the fense on what hardware to get as I still want something for home use but with fast fiel access. I could get a DS1511+ but thought it would be cheap and get better performance buildin gmy own custom nas device. I already have servers taht run hyper-v and esx 4.1 and would like to have shared storage and a large file share for my movie media.

 

[Pitbull]

I was just giving my opinion, as where to save money on in this build, if money is no object throw a server quality hardware at it, but i was assuming this was for a cheap home build, as people started suggesting micro atx boards.

 

[leeleatherwood]

Is using a zraid/zfs comapred to a traditional raid 5 still the preferred method? I read somewhere that an issue with zraid is that the IOPS performance is only the amount of a single drive and not the whole array, is that correct. So, does L2ARC help improve IOPS?

Aside from using the NAS for media and movie streaming, I woud like to host ESX vms. Am I better off with a raid 5? I am still on the fense on what hardware to get as I still want something for home use but with fast fiel access. I could get a DS1511+ but thought it would be cheap and get better performance buildin gmy own custom nas device. I already have servers taht run hyper-v and esx 4.1 and would like to have shared storage and a large file share for my movie media.

ZFS is better than Raid5 in every single way. The most beneficial feature of RaidZ is the dynamic stripe width and how it negates the Raid5 write hole. Basically since Raid5 does not have a dynamic stripe width, it must read whatever is in the current stripe, determine parity and then write.

With ZFS it does not have to read before it write, massive performance benefit here.

I dont think Raid5 has faster IOPs than RaidZ, if anything they are probably the same. On the other hand, RaidZ can make use of Ram and/or an SSD to cache small randomly accessed files, thus RaidZ has a huge performance advantage over Raid5 in terms of IOPS.

 

[danswartz]

In general though, if you need the IOPS for reads instead of writes, raid10 is better than raidz, especially for small random reads.

 

[leeleatherwood]

In general though, if you need the IOPS for reads instead of writes, raid10 is better than raidz, especially for small random reads.

That too, and higher spindle speed.

If IOPS if your priority then you probably dont need much data capacity. Therefor a Raid 10 or 0+1 would serve you better.

 

[dreadwing]

Interesting stuff. I was comptemplating if I shoudl just add a SSD drive for esx vm's to get the high iops. I use a ton of storage for media, so that is goign to occupy the vast majoirty of the nas device.

 

[brutalizer]

Here is a M1015 for $85: http://www.glcomp.com/ibm-system-x-serveraid-m1015-sas-sata-controller

I live in Europe, and see the price is $125. Are there different prices for Europe citizens and American citizens, or have the price been increased?

 

[s0rce]

I live in Europe, and see the price is $125. Are there different prices for Europe citizens and American citizens, or have the price been increased?

says $125 for me too, I swear it was $85 when I looked at it last. Maybe I was crazy. Sorry.

 

[dreadwing]

Price increased, as it was $85.00 when I first looked. Damn inflation is running wild.

 

[loosh]

I can not find the Intel SASUC8I for less than $153 on Ebay. What's another cheap RAID card that's as good?

i get mine for under $150 from provantage.

http://www.provantage.com/intel-sasuc8i~7ITSP0NQ.htm

 

[davros123]

Interesting stuff. I was comptemplating if I shoudl just add a SSD drive for esx vm's to get the high iops. I use a ton of storage for media, so that is goign to occupy the vast majoirty of the nas device.

yes, but large ssd's are expensive, and you'll need another backup solution - which having it on the zfs San provides via snapshots.

I do this for my win vm's as they do alot of disk access.

 

[dreadwing]

Thanks. What specs would you provide for mobo, cpu and ram.

$330[/URL] - NORCO RPC-4220 4U Rackmount Server Case

PSU wise, I recommend this:
$106 - SeaSonic M12II 620 Bronze 620W Modular PSU

 

[Dangman]

Thanks. What specs would you provide for mobo, cpu and ram.

If the Core i3 2100T does work with the X9SCL-F, then I recommend that route. RAM wise, Supermicro mobos tend to be a tad finicky when it comes to non-ECC RAM support. So I recommend this Kingston ECC RAM:
$80 - Kingston 2 x 4GB KVR1333D3E9SK2/8G ECC Unbuffered DDR3 1333 RAM

With the above RAM, the entire CPU + mobo + RAM setup is around $400 or so.

But if you don't really need super low power parts, Intel NICs, server quality, IPMI 2.0, KVM over IP, or ECC RAM support, and only plan on using one SATA controller, you might be fine with this:
$75 - AMD Athlon II X3 445 CPU
$42 - G.Skill Value Series F3-10666CL9D-8GBNT 2 x 4GB DDR3 1333 RAM
$91 - Gigabyte GA-880GMA-USB3 AMD 880G motherboard
---
Total: $208 plus tax and shipping.