Newbie ZFS / ESXi Build

N

noise850

n00b
Joined
Dec 9, 2005
Messages
22
I'm completely new to ZFS, ESXi, and Virtualization, but have been wanting to learn for a while, so I figured it would be best to put a system together. The purpose will be to run ZFS on a VM as well as a few other VMs, which would access their data through ZFS. I also want it to run as a NAS for our home.

There will be a lot of audio, video, and image data as my wife does graphic design and I do audio and video editing. For now I'm just going to start with GigE, but may want to go with a higher speed option in the future if the need arises to source HD Video. Other than that, just basic office documents and the such. The ultimate goal is to consolidate all of our files into one place so they are not spread out.

Power usage and heat is a concern, as I live in LA.

Parts list:

Case: Fractal Design Mini

Mobo: SuperMicro X9SCM-F

CPU: Xeon E3-1240 V1 or V2
I was wanting to go V2, due to the lower wattage and heat, but read that the X9SCM needs a firmware update, and I wouldn't have a V1 to put in to do that before I put in the V2. What do you guys think?

RAM: 4x8G Kingston ECC KVR1333D3E9S/8G

RAIDZ2: 6x 3TB 5k3000
I know these are old and discontinued, but I may be able to get a hold of some retail boxed 0S03230. I was planning on buying 2 more for spares, so 8x total.

PSU: Seasonic X650 Gold
I have this sitting in the closet unused. Will it be enough to power the system? I figured with only 5400rpm drives and no video card it would be fine.

HBA: LSI 9260-8i

Boot: Some sort of SSD, undecided yet.

So, what do you think? Should I go with the older 5k3000 even though availability is an issue? The concern is how WD will handle warranty claims. Also, I was thinking of buying another 4 of them for backups. Is this the best option? At work we use LTO5 with LTFS for backups, but I certainly can't afford an LTO5 drive. At home, I have some files randomly backed up on externals, but again it's an unorganized mess.

What about the Ivy vs. Sandy Bridge issue? I'd really like to use Ivy due to the heat savings, but I don't see any way around it as I don't know where to get a mobo with the latest firmware.

Any ideas on the Boot drive? I was thinking an SSD due to power and heat and I also was reading over using it for write caching. Again, I'm completely new so any advice is welcomed.

Thanks!
 
drescherjm said:
Do you really need that? You will not be using its RAID if you use ZFS.
Click to expand...

True, it's a bit overkill right now. My reasoning was if in the future I max out my ZFS system I could put this HBA into my main editing rig and do hardware raid, and buy a larger card for ZFS. But, who knows the future, I may just need to think about right now and go with a lower end card.

Any suggestions on what would work with 3TB drives? I've read some older cards have issues with 3TB, so again I figured a newer card would be more compatible.

Thanks for the help!
 
I would go for the IBM M1015 as it is cheaper, $80 on ebay. And this card does not have issues with 3TB disks, nor 4TB disks. They work fine.

If you need many IOPS, then it is mirrored pairs of disks. So, three mirrors, or four mirrors. And one disk for hot spare.
 
You do realize that with ZFS you do not need to use the exact same drive right? Unlike Hardware RAID, ZFS lets you mix and match drives, so later on you can replace a 5k3000 with a western digital and it'll be just fine. The only rule is that the replacement needs to be the same size or larger, you cannot use a smaller disk. No need to worry about buying extras or availability.
 
Well, you guys are definitely right, I'll drop the 9260 and get a M1015. See, that's why I asked!

So, what do you guys think about the PSU, will 650 handle it all?

Also, any thoughts on the SB and IB issue with the mobo? I'd really like to go IB since the cost difference is inconsequential, but I don't know how to guarantee I get a newer rev BIOS. Are there any issues with ESXi with the newer BIOS also?

Thanks again for the help!
 
Update:

I went ahead and bought an M1015 with cables and bracket on ebay, so I'm definitely dropping the LSI 9260 from the build. And thanks for the PSU advice too!

So, all that's left is to figure out about the E3-1240 vs. 1240v2 issue and what I'm going to do for a boot drive.

Do you guys think I should just say screw it and go with the Sandy Bridge 1240 and not worry, or is there a viable option to go 1240v2? Keep in mind that unless someone in LA wants to let me borrow a 1240v1 to use to flash, I have no real way of upgrading an older rev BIOS to the newer one.

Thanks again for all the help! This is why Hardocp has always been my #1 site, even if I mainly just quietly lurk and learn.
 
Regarding BIOS, the newer iteration of mobos are sometimes v2.0. Check with your reseller or ask supermicro which version of bios it is.
 
So, I pulled the trigger and ordered all the stuff.

I ordered the X9SCM-F-O board, 32 gigs of Kingston ECC mem, a 1230v2, and an 8 gig usb drive.

I picked up an M1015 on ebay, and the order for the 5k3000s is pending (hopefully it goes through)

So, anything else I need? I did pickup a 256gig M4 during the Amazon deal that's on the way. Should this be enough for the boot drive? Would I be able to also use it as a cache? Or should I save it for a workstation build I'm planning and go with something else for the server?

Thanks again for the help!
 
noise850 said:
? I did pickup a 256gig M4 during the Amazon deal that's on the way. Should this be enough for the boot drive? Would I be able to also use it as a cache? Or should I save it for a workstation build I'm planning and go with something else for the server?

Thanks again for the help!
Click to expand...

For a ESXi bootdisk with a base ZFS VM a 40GB SSD would be enough.
Using part of the SSD as a cache is not suggested. Best is to use always whole disks.
In case of a ESXi boot disk you might create a virtual disk so its also quite slow.

Save it for your workstation.
 
So, after a few weeks working (only had a little time on the weekends), I got everything, put it all together, installed ESXi, OI, Napp-It, and WHS2011. While I am learning a lot getting everything setup, I have ran into the same write caching issues as many others.

Right now, my windows workstations are just connecting using SMB. WHS2011 is connected to a NFS datastore on the zpool, and also has a storage drive connected through iSCSI on the zpool. Read speeds are super fast, averaging between 650-750MB/s. Write speeds are super slow, averaging between 10 and 15 MB/s. I just tried disabling sync on the pool, and instantly my writes jumped up to 550 MB/s.

So, what should I do? Do I need to add an SSD as a ZIL? How much will that increase the writes? I really only care about saturating a single GigE for the workstations, but it would be nice for the VMs to have faster writes. Should I just disable sync on the VMs and their storage and just keep it on for the important files? Keep in mind this is an all-in-one and it's on a UPS that can last 30 minutes.

Thanks again for the help! Once I get time and get everything together I'll post some pictures in the show off thread.
 
You only have a single raidz vdev using 5400 rpm drives?

That would explain the slow performance, especially for small files.

What exactly did you do to test write speeds? via nfs? via iscsi?
 
What Kingston 4x8gig memory sticks did you get? I ended up going with 4x4 gigs 'cause the 8 gigs were too expensive for me so I settled for 16gigs for $120. That model number you listed for the memory in your original post looked like the 8gig kit (2x4). Can you post the link to what you got?

I went ahead with the 1230 v2 and hope that the X9SCM-F board I got from Newegg has the latest bios.
 
gmui said:
What Kingston 4x8gig memory sticks did you get? I ended up going with 4x4 gigs 'cause the 8 gigs were too expensive for me so I settled for 16gigs for $120. That model number you listed for the memory in your original post looked like the 8gig kit (2x4). Can you post the link to what you got?

I went ahead with the 1230 v2 and hope that the X9SCM-F board I got from Newegg has the latest bios.
Click to expand...

I bought 2 of the Kingston kits off Newegg, and spent way too much on them. They are deactivated there now, but I also saw the mem at Provantage for cheaper than I paid. Provantage doesn't have the kit, but sells the individual sticks.

It's funny, I got lucky and got a rev 2 bios board, but there was no indication it was that until I powered the system on. The manual and the actual board itself both said rev 1.11a, so I figured it had an older bios. I had to go to microcenter to pickup stuff for the wife's system, so I bought a G620 for 40 bucks and put that in to flash...but the bios was 2.0, so that was a waste. I guess it just comes down to luck, either you will get a newer bios, or you will need a sandy chip to flash.
 
patrickdk said:
You only have a single raidz vdev using 5400 rpm drives?

That would explain the slow performance, especially for small files.

What exactly did you do to test write speeds? via nfs? via iscsi?
Click to expand...

I tested via nfs and via iscsi using crystal disk mark on a WHS2011 VM. With sync enabled, read was 650-750 MB, write was 10MB. With sync disabled, read was the same, write was 450 to 550.
 
noise850 said:
I bought 2 of the Kingston kits off Newegg, and spent way too much on them. They are deactivated there now, but I also saw the mem at Provantage for cheaper than I paid. Provantage doesn't have the kit, but sells the individual sticks.

It's funny, I got lucky and got a rev 2 bios board, but there was no indication it was that until I powered the system on. The manual and the actual board itself both said rev 1.11a, so I figured it had an older bios. I had to go to microcenter to pickup stuff for the wife's system, so I bought a G620 for 40 bucks and put that in to flash...but the bios was 2.0, so that was a waste. I guess it just comes down to luck, either you will get a newer bios, or you will need a sandy chip to flash.
Click to expand...

Hopefully I get just as lucky with the bios version on the board.

For the memory, are these the same as what you got?

http://www.amazon.com/Kingston-ValueRAM-KVR1333D3E9SK2-8GI-Memory/dp/B004S1OFJY/ref=pd_ybh_1

I'm debating whether to spend $200 for 2 of these vs. the $120 I paid for 4x4gig from Newegg. $80 more for the same 16 gigs but then have room to expand...
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
The stick you posted from Amazon is not the one I got. That one is a 2x4gb kit. I got 2 of the 2x8gb kits. Note the K2/8G at the end of the model number on the Amazon one, that's how Kingston states it is a 2x kit with the total of the 2x at 8 gig.

Here is the one from provantage:

http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

The provantage one says K2/16G, so its a 2x with the total at 16. Get one of those, and you'll have 16 gigs with 2 slots open to expand.
 
encore2097 said:
I'll let you in on a little secret: http://www.superbiiz.com/detail.php?name=D38GRE133S

Samsung 8GB ECC UDIMM 1.35v @ $81 / ea
Click to expand...

Thanks! Looks like I'll return my Newegg memory and get 2 of these instead.

But Supermicro lists a different model number under their list of tested memory. From their site it shows MT9KSF25672AZ-1G4M1 but the link you posted has this: M391B1G73BH0-YH9 1.35V.

Do you have this board and memory working together?

Thanks again!
 
noise850 said:
The stick you posted from Amazon is not the one I got. That one is a 2x4gb kit. I got 2 of the 2x8gb kits. Note the K2/8G at the end of the model number on the Amazon one, that's how Kingston states it is a 2x kit with the total of the 2x at 8 gig.

Here is the one from provantage:

http://www.provantage.com/kingston-technology-kvr1333d3e9sk2-16g~7KIN91M7.htm

The provantage one says K2/16G, so its a 2x with the total at 16. Get one of those, and you'll have 16 gigs with 2 slots open to expand.
Click to expand...

Thanks for the catch and link!
 
You must log in or register to reply here.
Back
Top