Possible ZFS file server upgrades

M

moose517

Gawd
Joined
Feb 28, 2009
Messages
640
Lately i've been noticing that my file server is starting to falter when it comes to my VM storage and was looking for some upgrades and was wanting some suggestions. Wanna keep it fairly cheap but if you can convince me to spend more because its truely gonna be worth it i'm all ears.

Anyways current hardware:
Intel C2D e6700 3.2ghz
Supermicro MBD-X7SBA http://www.newegg.com/Product/Product.aspx?Item=N82E16813182144
8gb ECC DDR2 800mhz ram
Norco 4020 with assorted HDD's(filled at this point)
2x supermicro AOC-SAT2-MV8 controllers

Currently the server runs solaris express 11 and i have 2 pools, one for our standard file storage, and the other on 4 seperate drives thats used for NFS for my VM's on another machine.
The HDD's fo the VM's at any given time if i look at the indicators are pretty solid such that the drives are more or less being accessed at any given time.

I was thinking of upgrading the motherboard to maybe 1155 with a CPU/mobo that can support VT-D so that i can run ESXi on it as well and virtualize OI also gonna need ram(prefer ECC, i'm great at corrupting things haha). also would like to add a SSD for cache(need ideas for this) and maybe a controller + sas expander for my main drives and another controller for the VM drives only. controllers need to support larger than 2TB drives as well.

EDIT: forgot to mention, one of the VM's runs a mysql server if that makes any difference at all because of the different I/O types it has
 
I have a similar out dated C2D E6750 w/ 8GB's of ram that I just swapped to a Q9550 to get me by while I look for a new setup myself.

I have been eying this deal: http://www.newegg.com/Product/ComboDealDetails.aspx?ItemList=Combo.890841

SUPERMICRO MBD-X9SCM-F-O and Xeon E3-1230 1155 CPU that supports vt-d for $379, so far I am finding that tough to beat. It's moderately low power @ 80W, so that's a plus too.

The only downside is RAM. It requires ECC UDIMM's, which are fine up to 4gb... so if you only need 16GB's, you're fine. However, if you want 32GB, I found the 8GB ECC UDIMM's to be a little more spendy.

If you can avoid SAS expanders, I would try, especially if you're going for VM performance. The IBM M1015 SAS Controller is pretty hot these days as an affordable card (~$65/ea shipped on ebay) that would get you 8 ports per card. The card offers >2TB support.

SSD's will vary,and we'd need more information about your size/usage and what you're looking for. As a very broad view, the Samsung 830 and Crucial m4 are pretty hot for the price vs performance right now for a read cache. I found recently with my setup as an NFS datastore, I was more gimped by not having an SSD write cache more than a read though... something to think about.
 
Last edited:
that deal is awesome, it's over 100 eur more expensive in germany ...
 
interesting, that motherboard is one i've eyed on newegg as well as that processor. Might just have to do that. Not decided if i wanna just go with 16gb or 32gb of ram, would i be better off going for 8gb dimm's, even if i only get a pair to start?
Considering that motherboard has enough PCIe ports i think i might just go with a few controllers, been looking at those M1015 cards as well so i think that sounds like a solid plan.

For the SSD's i was thinking of creating a ZFS pool of like 2 or 3 in a raidz purely for VM storage, and then another SSD to act as cache for the rest of the pools(if thats possible). Need about 700gbs of SSD storage though which might get expensive.
 
moose517 said:
Not decided if i wanna just go with 16gb or 32gb of ram, would i be better off going for 8gb dimm's, even if i only get a pair to start?
Click to expand...

If you do, when I was looking I found the Samsung DDR3-1333 8GB/1Gx72 ECC (M391B1G73BH0-CH900) @ SuperBiz: http://www.superbiiz.com/detail.php?p=D38GE133S2 $129.99.

It is ECC Unbuffered, and the part number M391B1G73BH0 is on the Tested Memory / compatibility lists from Supermicro on the MBD-X9SCM-F-O board.

$129 for 8GB is still higher than I'd like, but it's better than the $200 - $400 I've seen elsewhere.
 
well got all my parts in, went with 3 M1015, that supermicro board with an E3-1220 with 16gb of ram. Gonna do the hardware swap thursday and want to make sure of a couple things first off.

1) Should i go ahead and virtualize and create an all in one? i already have a dedicated box for my VM's and was wondering if you seen any reason to run like ESXI or just run solaris like i currently do( thinking of switching to OI actually)
2) pools must be exported i understand that, but how difficult will it be to import them back into the OS being on new cards? will it figure that out or am i gonna spend hours trying to reimport my storage drives?
 
1) There is long thread here, created by me, where we discuss ESXi or running bare metal. It seems as ESXi is stable enough, however I dont like that. Search for my name and the threads I have created for an discussion.

2) Just export the zpool and then import it. It takes literally seconds.
# zpool export tank
on the new computer: # zpool import tank
And then you are done. This is ZFS. :)
 
brutalizer said:
1) There is long thread here, created by me, where we discuss ESXi or running bare metal. It seems as ESXi is stable enough, however I dont like that. Search for my name and the threads I have created for an discussion.

2) Just export the zpool and then import it. It takes literally seconds.
# zpool export tank
on the new computer: # zpool import tank
And then you are done. This is ZFS. :)
Click to expand...

According to his post he is running on Open Solaris 11 and wants to move to Open Indiana. Open Solaris uses a higher pool version number so he won't be able to export and import it into Open Indiana.
 
Suprnaut said:
According to his post he is running on Open Solaris 11 and wants to move to Open Indiana. Open Solaris uses a higher pool version number so he won't be able to export and import it into Open Indiana.
Click to expand...
He is using Solaris 11 Express which uses ZFS version... XX

And OpenIndiana uses ZFS version...YY

If YY >= XX then he can just export / import. else he needs to move the data off to disks, and then reformat his old disks using OI. There is no other way.

Can someone clarify? Does Solaris 11 Express and OpenIndiana share the same ZFS version?
http://en.wikipedia.org/wiki/ZFS#Solaris_11
 
brutalizer said:
He is using Solaris 11 Express which uses ZFS version... XX

And OpenIndiana uses ZFS version...YY

If YY >= XX then he can just export / import. else he needs to move the data off to disks, and then reformat his old disks using OI. There is no other way.

Can someone clarify? Does Solaris 11 Express and OpenIndiana share the same ZFS version?
http://en.wikipedia.org/wiki/ZFS#Solaris_11
Click to expand...

OpenIndiana and all the free other options: max Pool V. 28
Solaris 11 Express: max Pool V. 31
Solaris 11: max Pool V. 33

If you use Solaris 11 and you do not need encryption, it is a good idea to stay on Pool V.28
even on Solaris. Otherwise you need to create a Pool V. 28 on Solaris and copy your data to this pool..
You can then import this V.28 Pool in OpenIndiana.
 
well this upgrade has turned from something that should have been easy to a pain. Got the new hardware installed no problem, booted right up. 9 HDD's weren't responding. Spent ages trying to figure out why, one of the HDD's even when in other locations won't come online so i know its bad(doesn't suprise me being a seagate from right around the time they ahd the firmware issues). the other 8 i'm fairly certain just means one of the controller cards is either bad, or needs reseated. Heres to hoping it just needs reseated.
 
Ran the failed seagate drive through seatools and its reports it as fully working. however solaris still doesn't like it. what can i do to re-add it to the pool?
 
moose517 said:
Ran the failed seagate drive through seatools and its reports it as fully working. however solaris still doesn't like it. what can i do to re-add it to the pool?
Click to expand...

You have a pool with a faulted disk.
Do a disk replace with a working disk.

There is no limit with number of controllers.
Do they all have the same firmware?
 
the disk is working, like i said i ran seatools on it on my desktop and it found 0 problem, ran the full thing. is there anything in the gui to do a disk replace or will i have to use the terminal? I'm assuming all 3 cards have the same firmware, bought them same time from the same person new.
 
When I upgraded to 1155 I was getting idle kernel CPU creep until the system would stop responding. I was running Solaris 11. Oracle only releases updates if you buy a support package, so I was unable to fix the problem. I had the exact same problem as you, my zfs version was 33, and I had to rsync everything to a new v28 pool, then install Openindiana. I really like openindiana so far, it is very stable.
 
geez, its sounding like i should just go back to my old hardware, get some new HDD's and attach them to the new board and copy everything to that. Such a hassle, simple hardware upgrade has knocked ~25TB of data offline and i guess into limbo as it currently stands.
 
anybody else running 3 M1015's on a MBD-X9SCM-F-O with solaris express 11? Having no music/movies etc is really driving me nuts :(
 
moose517 said:
anybody else running 3 M1015's on a MBD-X9SCM-F-O with solaris express 11? Having no music/movies etc is really driving me nuts :(
Click to expand...

I am running three M1015s on a X9SCL-F but with OpenIndiana (thus pool version is 28). X9SCL-F is basically the same as yours, even uses the same BIOS.

There seemed to be issues with the older BIOS, I just updated it this morning to the latest and greatest and it boots faster.

My M1015s are flashed to IT which was a pain in the ass but they also worked with their regular firmware but there was the also pain in the ass driver install with OpenIndiana.
 
i guess i could give updating the bios a shot, don't think it will help but if it makes it boot faster thats always a plus haha. Maybe i need to pull the controller and try it on my desktop and see if it works on it.
 
Anybody familiar with a chenbro CK13601 working with a M1015? i have googled and haven't come across anything confirming nor denying them working together
 
You must log in or register to reply here.
Back
Top