File server maybe ZFS

B

berrmich

n00b
Joined
Mar 7, 2011
Messages
59
I have a media center that I would like to build a file server for. It has a Ceton 4 tuner card and small SSD for a hard drive. I would like to be able to stream full bluray and still be able to record from 7MC. Also back up my documents, pics, ect. I do have a gb network. Right now I only have one TV but that may change.

My thought was this:

Intel xenon x3440 vs Intel Core i3
Supermicro X8SI6-F
Kingston 2x2gb ECC Unbuffered DDR3
5x Samsung Spinpoint F4 vs Hitachi 5K3000
Norco RPC-450 case
Do I need break out cables?


I was thinking of ZFS? Is there a good enough tutorial for someone completely new to BSD?
I would like some protection if a drive fails but wouldn't be the end of the world if I lost all my movies.

Will this be fast enough for recording live tv and streaming Bluray?

Which processor? What about samsung vs. Hitachi for this setup?

I've been reading this forum all week but I'm new to the server thing.

Thanks for the advice!

-Mike
 
You can use my ZFSguru project based on FreeBSD 8.2+ and FreeNAS 0.8 which is also based on 8.x. The other option would be something based on Solaris like napp-it and NexentaStor. Probably ZFSguru is the easiest option, but also the one with the least functionality.

I recommend you to try out a few of these NAS-specific distro's and compare them inside a Virtualbox VM just to see if you like the interface and it can do what you want.

Regarding your hardware, especially if you go FreeBSD you would have higher performance if you opt for 8GiB DDR3 instead (2x4GiB). If you want to start with 4GiB, i would recommend buying a single 4GiB module instead, for better upgrade ability. The performance difference from 4GiB to 8GiB is large enough to justify the expensive, i think. It also gives you the ability to upgrade to 16GiB later which would be useful if you add more disks.

Without encryption, compression or de-dupe, your CPU doesn't have to be that powerful; having lot's of RAM is more important here!
 
upgrading the ram is cheap. I can just start with 2x4 or even up to 16 if you think it would make a difference.

What features would I miss out on going with zfsguru? I don't mind if it's more challenging to set up.

So you think the i3 would be just fine then? Any real advantage to the xenon? It's only 100 bucks more or something like that.

-Mike
 
16GiB for just 5 disks in RAID-Z would be excessive, but i think you still would have a performance benefit. Consider your NAS has some data that you use everyday. Say a firefox profile and other stuff you access everyday. Since your NAS is running, ZFS could figure out quite well which stuff is frequently accessed. ZFS caches recently accessed and frequently accessed data separately. So alot of memory could mean you end up reading alot from RAM from the stuff you actually use the most. RAM of course, is the fastest of all.

A decent setup now would be 8GiB RAM and upgrade to 16GiB when you add more disks. However, consider that DDR3 prices may be climbing again and you could opt to buy more RAM now. The CPU doesn't need to be that powerful, but try to get multiple cores; ZFS loves cores! It can use ALL of them. So a low-clocked 8-core would be much better than a high-clocked 2 core. As for Intel you could opt a 4-core if price allows. Don't bother about HyperThreading you better turn that off for performance reasons. If you like you can test performance with and without. One useful feature would be AESNI hardware accelerated encryption if you're using that. FreeBSD support this through GEOM and works great.

But the core i3 would not allow usage of ECC function, so the Xeon has a benefit here. Some Xeon's also have vt-d which is important if you want to use ESXi, but not for a normal plain ZFS install. It kind of depends on your situation, if you value performance then try to avoid as many pitfalls. You can also opt to keep the system cheap and upgrade some time later when perhaps Bulldozer is out, an 8-core bulldozer would be extremely cool. :D

Word has it that Bulldozer will outperform Sandy Bridge on IPC (clock-for-clock). Either way, i have to be fair, this is totally out of line with what is needed for a simple 5-disk RAID-Z box. But if you like me want to switch to 10Gbit one day, use fast SSDs to accelerate performance and such, it can be quite tempting to buy overpowered hardware, in an effort to exclude as many potential bottlenecks that might prevent the desired performance level.
 
Ecc will work on the i3's with motherboard support (not on i5 or i7 though) - unbuffered only though
Ref:http://www.supermicro.com/support/faqs/faq.cfm?faq=10712

You might look at a Dell T110 - 330US will get you a x3440, 2gb ram, case, mobo, ps. With one 5.25 to 3.5 adapter you can mount 5 drives internally. Add some ram from crucial and you are pretty much set. Main downside is expandability - adding any additional drives will require an external case.

I'd go with the x3440 if then extra hundred is in your budget, but you can probably get by without it.
 
Hi - I built a system very close to this for testing at work and will duplicate it at home once I get the configuration right.

I used the core i3. I was having system loads of 4 and higher when copying large files (20GB+) to the server. I was testing it as a VMWare datastore using NFS and running Nexenta Core and Napp-it. I found out that when I created the raid-z pool of 4 7K2000 2TB drives I had compression turned on. When I turned compression off the system load did not go over .5.

I used the Norco 4220 chassis because I am expecting to add many drives.

The X8SI6-F board comes with two forward breakout cables. If you want to go from an SFF-8087 back plane and use the SATA connectors on the MB you will need at least one reverse breakout cable.

I am going to continue testing by adding an X25-E SSD for ZIL and upgrading the processor to an X3440. If the performance does not satisfy for enterprise database workloads then I may try some SAS drives to see what happens.

Hope that helps.
 
Just a quick note: the advice to buy an SLC SSD instead of MLC is outdated and does not apply to current SSDs anymore, which all use write remapping. People used SLC SSDs for ZFS SLOG function because the simple SLC NAND controllers do not use write remapping. This is the first generation of SSDs that were popular in 2008/2009. Modern SLC versus modern MLC does not have many benefits; though you do have higher write speeds of course.

If you have an SSD without supercapacitor, you should use it as either primary storage or L2ARC, but it would not be safe to use it as SLOG/ZIL device. All modern SSDs do write remapping and thus are vulnerable on sudden power cycles.

If you have an SSD with supercapacitor (Intel G3, Marvell C400, some Sandforce SF2000 enterprise models) then it would be suitable for SLOG as well. The G3 also has better write speeds than the G2 models.
 
Chris: The T110 is on sale but the to stay around 300 is with a celeron processor. Jumps to 628 once you add the x3440. I think I can do much better for the same price.

Sub.mesa: I'm assuming I don't need a very big SSD drive? Do you have one in mind that would be good? Or do I just run it off a USB-key?

Looks like the x8si6-F is out of stock on newegg. Anyone purchase on ebay?

Hard drives... any problem I will run into with the 5k3000 drives and this setup? Should I just get the 7k3000s for only 30 bucks more?

Drake3: Thanks, Let me know how your testing goes. I'm very interested. :)
 
Personally i would wait with buying an SSD until the new Intel G3 comes out; would be perfect since you can use it for everything:
- hand off VMs or other performance-sensitive data on primary storage or ZVOL
- use it as L2ARC cache
- use it as SLOG/ZIL

The performance of Intel G3 should also be very good, particularly on random writes where they should be improved quite a bit. I also hope fragmentation and performance degradation will be less than G2, but i've yet to see real thorough benchmarks and those probably will not available until either shortly before or after the release.

Hitachi 5k3000 should be good, if that's the one that uses 666GB platters? I would not use 7200rpm anymore; 5400rpm does sequential I/O very well and random I/O you can do with SSDs much better than any HDD ever could. Especially for ZFS it makes sense to run high density but low rotation speed disks (4200rpm) - coupled with SSDs to ensure the HDDs only have to do sequential (contiguous) I/O. If it all works out, it would get the best out of both technologies (SSD and low rpm HDD) while at the same time doing that at very low power consumption. 5400rpm drives you don't have to actively cool if they have some breathing space, which is a distinct advantage as well.
 
About the X8SI6-F; it has a nice SAS2008 onboard controller and server chipset so Core i3 + ECC would work as i've been told, but it doesn't have alot of PCI-express.

One alternative would be this board:
http://www.supermicro.com/products/motherboard/Xeon3000/X58/X8ST3-F.cfm

Has an older LSI 1068E onboard but more PCI-express slots (4). It does have a desktop chipset however, so Core i3 would not have ECC functionality on this motherboard if i'm correct. If you choose a Xeon CPU you should have ECC on this board and it makes a reasonably cheap investment for all the functionality offered.
 
You can't use i3 at all on that mobo. It is x58/1366 based so its i7/x55xx/x56xx only.
 
Pardon me, you're right. It's socket 1366.

SuperMicro has some newer boards too for sandy bridge platform, but no product pages yet.
 
As sub.mesa points out the X8SI6 only has two PCIe slots, x4 and X8 available. Since I am using a chassis of 20 drives I didn't think this would be a big issue. I can connect 8 drives to the onboard LSI 2008, buy another controller for 8 more drives and connect the remaining four to the onboard SATA ports using the reverse breakout cable. I would still have room for another PCIe board, probably another dual port network card. Well, that is the plan anyway.

So I have two vdevs setup right now. 4-drive raid-z in the first pool, 2-drive mirror in the second pool. All Hitachi 7k2000 2TB, Core i3, X8SI6, 4GB ECC, Nexenta Core.

I used a Windows 2008 server physical machine for benchmarking using CrystalDiskMark connected via single GBe link. I tested using SMB and NFS. I am not sure how good the NFS client is in Server 2008, but it was significantly slower than SMB. I don't have any NFS tuning expertise. I figured Nexenta(Solaris) would rock with NFS. It is surprising to me.

Nexenta 1 - zpool1 - raidz - 4 7k2000 hitachi - SMB
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 60.939 MB/s
Sequential Write : 92.483 MB/s
Random Read 512KB : 61.320 MB/s
Random Write 512KB : 87.854 MB/s
Random Read 4KB (QD=1) : 14.261 MB/s [ 3481.7 IOPS]
Random Write 4KB (QD=1) : 12.134 MB/s [ 2962.5 IOPS]
Random Read 4KB (QD=32) : 113.194 MB/s [ 27635.4 IOPS]
Random Write 4KB (QD=32) : 86.372 MB/s [ 21087.0 IOPS]

Test : 1000 MB [Z: 41.8% (2285.5/5466.7 GB)] (x5)
Date : 2011/03/15 12:06:01
OS : Windows Server 2008 Server Standard Edition (full installation) SP2 [6.0 Build 6002] (x86)

Nexenta 1 - zpool1 - raidz - 4 7k2000 hitachi - NFS
-----------------------------------------------------------------------
CrystalDiskMark 3.0.1 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 46.818 MB/s
Sequential Write : 47.370 MB/s
Random Read 512KB : 46.842 MB/s
Random Write 512KB : 42.117 MB/s
Random Read 4KB (QD=1) : 16.199 MB/s [ 3954.9 IOPS]
Random Write 4KB (QD=1) : 14.312 MB/s [ 3494.1 IOPS]
Random Read 4KB (QD=32) : 16.266 MB/s [ 3971.2 IOPS]
Random Write 4KB (QD=32) : 12.097 MB/s [ 2953.4 IOPS]

Test : 1000 MB [N: 41.8% (2285.5/5466.7 GB)] (x5)
Date : 2011/03/15 17:19:53
OS : Windows Server 2008 Server Standard Edition (full installation) SP2 [6.0 Build 6002] (x86)

I was close to maxing out the ethernet connection with Crystal on the sequential write. I did max out the connection with ATTO Disk Benchmark.

I am waiting for a new switch for my test rack so I can LAGG the network connection to the zfs test box.

But that should give you an idea of the performance with a simple raidz pool.

If anyone has any ideas why NFS was so much slower please let me know. I would like to use NFS to connect the zfs machine as a VMWare datastore and would like to max the performance for the hardware.
 
I believe iSCSI on Solaris is always sync writes; that would be slow without SLOG SSD. You should read some Solaris-specific NFS tuning stuff that allows you to ignore these flush commands via NFS. You can also try changing the options on your client to only use local file locking, if that would make any difference.

NFS performance from Ubuntu linux to ZFSguru is very good over here. By the way your Samba scores look very good; especially that multithreaded random I/O allows performance scaling; very good!
 
berrmich said:
Chris: The T110 is on sale but the to stay around 300 is with a celeron processor. Jumps to 628 once you add the x3440. I think I can do much better for the same price.
Click to expand...

It seems to matter how you navigate to that page, try this link:

http://configure.us.dell.com/dellstore/config.aspx?oc=bedwlr1&c=us&l=en&s=bsd&cs=04

Re: NFS vs. Samba - I saw similar speed issue Samba was 100-110Mb/sec, NFS (Win7 Ultimate client) was 40-45 Mb/sec on the same pool/filesystem. Solaris 11 Express / x3440 / 16Gb ECC.
I actually have a 4 disk raidz 5k3000 pool (one of the disks from newegg was doa, waiting on the return to merge it into the main pool). I'll run some benchmarks on that to toss up for comparison against the current 5 disk 7k3000 pool.
 
Last edited:
ChrisBenn said:
It seems to matter how you navigate to that page, try this link:

http://configure.us.dell.com/dellstore/config.aspx?oc=bedwlr1&c=us&l=en&s=bsd&cs=04

Re: NFS vs. Samba - I saw similar speed issue Samba was 100-110Mb/sec, NFS (Win7 Ultimate client) was 40-45 Mb/sec on the same pool/filesystem. Solaris 11 Express / x3440 / 16Gb ECC.
I actually have a 4 disk raidz 5k3000 pool (one of the disks from newegg was doa, waiting on the return to merge it into the main pool). I'll run some benchmarks on that to toss up for comparison against the current 5 disk 7k3000 pool.
Click to expand...

Thanks, that worked. :)

This is pretty tempting...

-Mike
 
sub.mesa - Is there a disk bench program you recommend to get a good random IOPS score from linux clients? I was just looking into Oracle's Orion tool, but never have run it or seen output.

Will your ZFSGuru pre-built system work with the LSI 2008 controller with the board I am using? I wanted to try FreeBSD next, but was not completely sure of the LSI driver status in FreeBSD.

Thanks.
 
What exactly do you want to benchmark? ZFS performance, iSCSI performance over the network? You mentioned Linux clients so that would imply testing performance over NFS or iSCSI from a ZFS server to a Linux client.

If you want to test sequential performance (copying large files) then the dd command can be a powerful benchmark tool. For most NAS sequential throughput is the most important spec (read+write). You should use dd on a filesystem though; if you test it on a raw device you may get unrealistically low performance.

As for testing ZFS performance, my ZFSguru dist has a nice benchmark built-in that produces graphs of varying zfs configurations. You can find it on Disks->Benchmark. This test is destructive so use on empty disks only. You can select which disks to benchmark. Generally you would get an error if you tried benchmarking a disk that was part of a pool, but better make no mistakes here. IOps performance on ZVOL is done with 'raidtest'. I did not incorporate L2ARC/SLOG devices in the benchmark yet; that feature will be available in a future release.

ZFSguru works with LSI SAS2008 controllers, but not those that work in IR mode only (like a IBM variant) - only IT mode firmware is supported. The driver is also not as mature as the older 3Gbps LSI driver, but i recently updated the LSI SAS2008 driver called "mps" to a newer version. This newer version is available in the recently released 8.2-002 system image, available for download inside ZFSguru (System->Install->Step 2). This system image is a stable release.
 
Ok.. here is what I have so far:

Intel xenon x3440
Supermicro X8SI6-F
Kingston 2x4gb ECC Unbuffered DDR3
5x Samsung Spinpoint F4
Norco 4020 case
Ordering corsair AX850


What else do I need? I'm using this mostly to serve my HTPC but also as backup to my desktops. I need enough speed to stream blu-ray content.

Do I need a separate system drive? Should I order an SSD for this? What about ZIL and L2ARC.. would this be that helpful for me for this use?

I have a copy (MSDN) of server 2008r2. Should I use this and virtualize ZFS? (I have no idea how to do this, but I can learn).

Sorry if any of these are stupid questions. I've never built a server.


-Mike
 
Also I noticed this limitation for ZFSguru: •Missing feature: Error recovery; replacing failed devices

What does that mean? What do I do when one of my disks dies? Or am I not understanding correctly.

I have started with 5 drives. I plan on added 5 drives at a time as needed (up to 20). Is this a good strategy?

Also I wanted to mention that I'm running a ceton 4 card and want to store my recorded TV on the NAS. I use Windows Media Center 7. With this in mind does running windows server 2008 give me any advantages?

I was also thinking of security cameras at home and at my shop. If I plan this for the future does that make any impact on my OS choice?

-Mike

edit: Sorry for the 2 posts I should have edited my first one. :) I just came up with more questions as I researched.
 
All it means is there is no support in the gui - you can still replace failed devices from the cli. If you are going to add 5 at a time, I'd suggest starting with a 5-disk raidz, and adding 5-disk raidz's to the pool as you expand it.
 
Ok.. is there anything I'm missing from my build? Should I put the OS on an SSD. What about this ZIL and L2ARC? How important will that be for me?

What I meant by asking about 5 drives... Is that a good number as far as speed/space? I can get 1 more and plan on 3x6 instead of 4x5? Or get 10 at a time?

What about windows 2008r2 with virtualization? good/bad?

- Mike
 
You must log in or register to reply here.
Back
Top