NAS - ZFS Build

[mR_CaESaR]

Hey Guys,

First post so, please go easy

I've decided to build a NAS as I've now had a couple of failed drives without any form of parity and it's starting to get annoying.

I'm just wondering if you guys can see any issues with my proposed setup:

- Gigabyte GA-Z68MA-D2H-B3 $AUD129 (the only reason I'm going for this is because I've read on another forum that they were able to successfully use their IBM M1015 on this motherboard, now I'm quite comfortable in getting the cheapest H61 based motherboard, I'm just worried that the M1015 won't work)
- Intel G850 $AUD75 seems like it'll be enough for my needs
- 8gb of ram
- Corsair 430wat PSU (enough power for
- I'm still tossing up between a bitfenix shinobi or a fractual R3, the shinobi is half the price and appears to have space for a couple more 3.5" drives, I'm just worried about the heat of the hard drives.
- HDDs I'll be using will be WD blacks. A friend of mine has recently upgraded his stroage from 1TBs to 2TBs and is letting go of about 15 drives for a great price. I'm personally looking at putting 12 or so drives inside the case (or as much as I can physically fit)
- IBM M1015
- FreeNAS 8
- RaidZ2

So here are now some questions:

- Will the CPU be enough for its purpose? The G850 is about $AUD75 and i3 costs about $AUD130
- Do you guys think it's worth spending almost double on the mobo? ($60 for a cheap all in one compared to $130 for the Gigabyte GA-Z68MA-D2H-B3 mobo listed) I don't plan on using another pci-e x16 card (only the 1x IBM M1015), I don't plan on expanding this NAS, if anything, just swap out the drives to bigger drives or if worst comes to worst, just build another cheap NAS with another M1015 (maybe one day once the prices of the drives come back down, i can have my 12x2tb array
- I guess depending on which mobo I choose will depend on me having the ability to go from 8gb to 16gb on the cheap. I know with RAM being so cheap these days, will I really see a benefit with getting 16gb or ram compared to 8gb?
- Do you guys think its worth spending also almost double on the case? ($70 for the shinobi, $140 for the R3), I've read a few reviews about the shinobi saying that it's front intake isn't anywhere near as good as the R3. Noise won't be a problem for me, the NAS will be in the garage hiding from everyone
- How hard is it to get the M1015 to work? I've now got the card (still waiting on cables and the rest of the build to test - unless I can test on a windows bases system?), I've read a few things about flashing it to IT mode, but I'm just wondering if it'll work with FreeNAS8 as stock IBM f/w? Also, did I get the right cable? MINI SAS 4i SFF-8087 36P To 4 SATA 7P cable?

Main purpose:

- It's just going to be my home theatre storage, purely to stream files to my htpc and network media tanks.

Anything else I've missed?

Any input will be greatly appreciated as this is all very new to me and I'm what you'd call a FreeNAS n00b

Thanks in advanced.

 

[mR_CaESaR]

Thanks for your reply Keljian, you've definitely swayed me quite a bit

I built something similar and live in Australia.

The main tip I'll give you is while the G series pentiums will be more than capable enough, the i3 has much better power saving features and, being that it will be spending a long time idling, you will save that money in power usage over the life of the server. If you are thinking about running multiple VMs, consider going up again to the base level i5 2320 or 2400.

Thanks for the tip, this is actually quite interesting and I might consider it.

We use ESX Hosts at work with VMWare vSphere and while I don't do the configs and vm builds, it would be great to be able to learn about building vm exchange/AD servers + also learn about disk management

I'm thinking this might be better as another build though?

I'll definitely look into some second hand i3's on OCAU.

I used an ASrock Z68M-USB3 - the microatx version, I bought it for 3 reasons,
1. it had vt-d support,
2. it had 6 onboard sata plugs so I could run "cheap" software raid until I could afford more.
3. it was cheap.

I ran into the challenge that I wanted more ram and it only had two slots. I worked around this by buying 2x8gig geil sticks for $140~ from Scorptec - but I now have 2x4gig sticks kicking around

I should have done more research and bought this board: MSI Z68MA-G45 - as it had more ram slots, and more pci-e slots, for a small amount more money and more time (would have needed to bought online). Gigabyte doesn't have a working Vt-D board.. otherwise I would have suggested going with them.

I don't think spending a huge amount more money on the motherboard would make that much difference.

Thanks for you're suggestion on the mobo. I'll definitely look at that mobo.

Just one question, is it the B3 or the G3? Know what's the difference between the two?

This mobo looks like it'll be great value, 2x pci-e x16, 4x ddr3 slots, 6x sata, can't really complain at that price.

I used a fractal design define mini case, it was pretty good to work in and has plenty of room for more drives if I want to go beyond my 6 (5 raid + 1 OS/VM). I can fit a 3-in-2 in the top 2 drive bays to increase the drive storage capacity to 9, and that is more than enough for me. Plus it doesn't take the space of a midi/normal tower. I got it for $125.

I'm still looking at a midi tower, ideally i would like to be able to fit around 11-12 hdd's and from what I've been researching, the two candidates seems to be the NZXT Source 210 Elite or the Bitfenix Shinobi. I'm really liking the R3 because of the way the hard drives get installed, the cable is behind and with the right length cable, you could effectively hide the sata cables, the only thing I'm kind of bummed about the R3 is i will lose space for 1 drive (whether or not this will be the make or break still remains as I haven't got all my parts yet). Looks definitely is not a factor as it'll be hidden away in the garage.. all I'm worried about is the temperature of the drives, all cases appear to be able to put 2x 120mm front intakes, 2x 120/140 top intakes, 1x 120mm rear exhaust and another 120mm bottom intake, this should keep the temps down surely?

I bought a Seasonic SII 2-430W 80 bronze from MSY for about $65, as it's quiet, cheap and relatively well reviewed, as well as being pretty efficient.

A few people have recommended Seasonic, I'll look into this more. Modular PSU's will be even better for cable management

I have looked into ZFS - and come to the conclusion that ideally you should have 1GB per TB of hard drive space after the first (eg if you have 3 TB of storage, you need 2GB of ram) this is only a rule of thumb. This is if you want to maximise the output bandwidth you have from the array.

This being the case, and ram being as cheap as it is, I recommend going at least 8 or 12, 16 I think is overkill - but if you're looking to spend money, get 8 gig sticks because it means you'll get more upgradability later (32gig anyone?).

Remember to take into account your bandwidth requirements though - eg if you're only looking to saturate a gigabit link, you probably don't need more than about 4gig of ram for the server, even if you're running 12 drives.

I know 8gb is probably way more then what I need for my purposes, I might as well go for the whole 16gb, ram is soooo cheap it's not funny.

This way if I also end up going the ESX route and install VMWare I won't have any issues.

For drives, I would look at going higher capacity if possible, as the more motors you have running, the more power you're consuming. Few drives with higher capacity will win in terms of power consumption over many low capacity drives. Remember this thing will be running 24/7, power is important. This goes double for faster drives- you don't need faster drives for what you're doing.

Would love to go high capacity straight up, but unfortunately I can't.

A friend is giving me a great price on his 1tb wd blacks (i know, more drives and more power), but this would be a quick way for me to end up with a 10tb raidz2 (if i use 12tbs) array.

The idea is, over time, I'd be able to get higher capacity drives.

I'm hoping over the year, I can change all the 1tb drives into 2tb drives.

Consider that you can get 3tb drives in external cases(which you can remove) from DSE for $169, then tell me you can get better value per gigabyte/power from a friend.

$169 for 3tb?! Have you got a link?

[Edit]
I would certainly consider running ESXi/Xen or KVM on the machine with passthrough of the drive controller to OI/nappit or FBSD/nappit or whatever setup you want, but in a vm. This opens up the things you can do with the machine beyond simple file serving and certainly, with either the i3, or i5 if you choose to go down that route, you will have the power to do that.

Forgot to mention - I bought an intel nic as I didn't want to stuff around with realtek onboard nics, whether you do this is entirely up to you

[/Edit]

I'm considering spending a little more to be able to run ESX on it. Thanks for the suggestion.

I'm still looking at controller and how it all works, I'm reading about flashing it to IT mode, or flashing it to LSI f/w, it's all relatively confusing at the moment.

I tried loading it in windows (doesn't help that I don't have my Mini-SAS to SATA cable yet) and this was the message I got:

And

From my research the above happens cos Gigabyte motherboards don't allocate memory to the PCIx16 slots if its where the video card would normally go.

I don't know.. it's all so confusing, so much learning

 

[ckryan]

FYI, while there are other reasons to consider the i3, the 1155 Pentiums and celerons (with the exception of the single core Celeron) idle just as well as i3's. I'm using a Celeron G530 2.5gHz in a low power system, and it doesn't idle any higher or any lower than other 1155 CPUs. The Single core 4xx series celeron is not what you'd want anyway.

If you can get a good deal on an 1155 i3, then get that -- there are some other reasons you might want to spend a little more and get one. If idle power consumption is your only concern, the dual core Celerons and Pentiums will work just fine too.

 

[dsumike]

$169 for 3tb?! Have you got a link?

HITACHI Touro Desk 3TB USB 2.0 Black External Hard Drive - $179 - $20 w/ Promo Code EMCNGNB39 = $159.00

One of the reviews states: "Inside is a brand new 7200rpm SATA drive for about half of what a bare drive is going for now."

 

[balance101]

Do you have 8x slot, works fine on my ud5

 

[mR_CaESaR]

FYI, while there are other reasons to consider the i3, the 1155 Pentiums and celerons (with the exception of the single core Celeron) idle just as well as i3's. I'm using a Celeron G530 2.5gHz in a low power system, and it doesn't idle any higher or any lower than other 1155 CPUs. The Single core 4xx series celeron is not what you'd want anyway.

If you can get a good deal on an 1155 i3, then get that -- there are some other reasons you might want to spend a little more and get one. If idle power consumption is your only concern, the dual core Celerons and Pentiums will work just fine too.

Interesting, I didn't know that. Thanks for the info. Still deciding on how to go about things...

Looking at an i3 but Keljan is swaying me for that i5 lol

Heya - first off all - I'm Jazper on OCAU (going to keep this straight to the point as I am strapped for time), glad I could assist.

Cool cool, I'll keep a lookout, I've been lurking the storage section for quite some time now just reading and reading

3tb drives as per this thread: http://forums.overclockers.com.au/showthread.php?t=1016090&highlight=3tb

Looks like they've gone up again at JB - check out the thread for other info.

Yep, looks like they have gone up, still at 200 a pop, it's still not bad

Re building another machine for ESXi/Xen etc - I don't see the point, this is where you'll have your key storage, so this is where you can dump vms a plenty, whereas you'd have to buy more hardware (processor, mobo, memory, case) and more drive space for another machine.

Yeah I can see your point, initial outlay will be a bit more, but in the long run I think I'll save more.

I'm just a little worried if I have my data on there and I screw it up... I'm scared that I'll somehow lose the array.. But I'm not too familiar with this stuff so I might be worrying over nothing

Re: processor

I would push to get an i3 or i5, plenty of power, and allows you to task more things to it than a lesser processor

I've been looking out for i3s or i5s on ocau, I think second hand socket 1155 is the way to go

Re motherboard: I meant the G3 motherboard - see this thread: http://hardforum.com/showthread.php?t=1675273

One note though, if you're dead-set on getting a normal tower case, then you will be able to find a 8 sata port motherboard for that price which fits the bill and has more pci-e slots (Make sure it has vt-d support though if you plan to go esxi, this is not as easy as it sounds).

I am space poor, and every cubic cm I can save is important, which is why I wasn't going to go any bigger than the fractal design define mini (which you can still hide sata cables with, however I have chosen not to in case I need to pull a drive due to failure - that way it is easier to remove them, airflow is not an issue with the two fans that come with the case). This space problem is also my reason for sticking with uATX, I had originally been looking at mini-itx.

Space is really no issue, I've got a section in the garage for the box, I've already got the ibm m1015 controller card so extra sata controllers really isn't that big of a deal. I just thought get an matx board just to make it easier, with the all in ones

How do I know if it's vt-d compatible? I can't see anything on the specs section that mentions anything "vt-d"

A modular PSU is not that important for what you're doing, most cases will allow you to tuck extra cables away, and you'll be using most of them anyhow.

Figured as much.. Currently got my eye on a seasonic psu on ocau

Do you need 10TB straight up or is this just something that you'd like? I have about 6TB worth of storage and have about 2TB free. The power consumption of an extra 4-5 disks is nothing to be sniffed at (5-10w per disk, 25-50w). Remember you can always add disks if you need to in a new zpool (as I understand it) so you could start with 5 and then add another 5 if you need them.

In my current jbod setup I've got about 5tb to migrate to the array, the extra 5tb was just for something to last the next year or so

I recommend fractal design - their cases are more expensive, but they are easier to work with.

You've turned this cheap 300ish build into a 600ish build lol...money is a little tight.. (mortgage, bills, toddler with another one on the way) so im only budgeting around 150 a month for the build (without disks as ive already sorted that out with my mate). I think though I'm better of saving a few more months then rushing it

HITACHI Touro Desk 3TB USB 2.0 Black External Hard Drive - $179 - $20 w/ Promo Code EMCNGNB39 = $159.00

One of the reviews states: "Inside is a brand new 7200rpm SATA drive for about half of what a bare drive is going for now."

Thanks for the link, but it makes it a little harder for me cos I'm in Australia

Do you have 8x slot, works fine on my ud5

Not on where I initially tried it, I'll have to try it on a few other pcs, I just didn't really fuss over too much about it as I don't have the cables yet

Thanks again for your help guys

Never ever again will I reply with quotes on an iPad lol... Took bloody long enough!

 

[brutalizer]

The risk of losing the array is much smaller if you run the thing in a VM and tinker in another vm for services, than if you run everything on metal..

Can you elaborate? Why is this true? I prefer running on metal, so it is more stable and less things can intervene and disturb.

 

[danswartz]

I agree, Keljian. Having standardized virtual hardware means fewer variables. On top of which, if you are going to make a bunch of potentially destabilizing changes, you can snapshot the VM before doing them - if you bugger it, you just rollback.

 

[tdg]

If the safety of your data is a top priority you may want to think about using a CPU/mobo that supports ECC, which means either moving up to a Xeon or to AMD. Running ZFS without ECC leaves you vulnerable to silent corruption during writes from bad memory.

 

[brutalizer]

Running ZFS without ECC leaves you vulnerable to silent corruption during writes from bad memory.

This is not unique to ZFS. This problem applies to ALL filesystems and storage solutions. The point is that ZFS immediately detects these errors, so it seems that ZFS is fragile when it reports errors. But other storage solutions does not even detect these errors, so you believe everything is ok, which is not. There are numerous stories on this. People change fileystem to ZFS, and suddenly gets error reports. Earlier, these errors where not detected.

 

[brutalizer]

I agree, Keljian. Having standardized virtual hardware means fewer variables. On top of which, if you are going to make a bunch of potentially destabilizing changes, you can snapshot the VM before doing them - if you bugger it, you just rollback.

Why would virtual hardware mean fewer variables? I like to eliminate as much as possible and run bare metal. There are several stories when people have used VirtualBox and Solaris, and VirtualBox behaved bad so the user lost data. There is a reason in production, you run on bare metal. How good does error messages in ECC RAM propagate to the VM, satisfactorily? I can mention several other things that I am not convinced will work without problems. When you want something to be stable, you typically eliminate as much as possible. KISS.

Regarding the snapshoting of VMs, I agree it is good. But I do that as well in Solaris 11, with ZFS. ZFS supports snapshoting the system disc, and if there is an error I just reboot into an earlier snapshot. I do have that feature in Solaris.

 

[dsumike]

Why would virtual hardware mean fewer variables

What they are getting at is if you use the system for other purposes as well rather than JUST a storage server, then virtualizing may make more sense. You will be able to logically separate your storage server from other services. Then, if you mess something up in one of your other VM's, blow it away and start over and your main VM was fine. Fewer variables to mess with the actual storage server itself.

The snapshotting becomes an extra layer of ease of making changes that is then outside the instance itself, which helps with the idea of KISS. It's a single click, done and just "works."

There are several stories when people have used VirtualBox and Solaris, and VirtualBox behaved bad so the user lost data. There is a reason in production, you run on bare metal.

This comes down to using the right tool for the job. VirtualBox has a bad enough reputation with being unstable and generally poor performance. Not to mention you are back to installing software on your storage server OS again...

Why use VB when a more enterprise-class software package is available for free? The free version of ESXi is generally more than enough for the average home user or even SMB user who wants to get up and running. Install ESXi bare metal and run your VM's on that.

Running VM's on ESXi has become very much of a "standard," especially in large data centers, with many people preferring this to running on bare-metal. There are plenty of other reasons this is nice, even down to standardizing what is presented to the OS, making changing out hardware a snap.

 

[danswartz]

Why would virtual hardware mean fewer variables? I like to eliminate as much as possible and run bare metal. There are several stories when people have used VirtualBox and Solaris, and VirtualBox behaved bad so the user lost data. There is a reason in production, you run on bare metal. How good does error messages in ECC RAM propagate to the VM, satisfactorily? I can mention several other things that I am not convinced will work without problems. When you want something to be stable, you typically eliminate as much as possible. KISS.

Regarding the snapshoting of VMs, I agree it is good. But I do that as well in Solaris 11, with ZFS. ZFS supports snapshoting the system disc, and if there is an error I just reboot into an earlier snapshot. I do have that feature in Solaris.

Well, I disagree. If you run production quality hypervisor like ESXi, it is extremely stable. As far as KISS, I believe you have it completely backwards - if you have more than one VM, the huge advantage of virtualizing, is that you can present *identical* hardware to every single VM, and you can make sure it is known good virtual hardware (meaning, no flaky realtek NIC drivers, etc...) As far as snapshotting, I'm aware you can do it in the guest, but not everyone does that. Or maybe you meant running on virtualbox in solaris? In which case, that is a point, but most people using virtualization in production are using xen or ESX/ESXi. Again, if you have more than one server, it is often impossible to disable all of the onboard "features", which can cause issues for some chipset/OS combos.

 

[mR_CaESaR]

You guys lost me at "run everything on metal"

I think I'm going to go down the vt-d route and learn about esx and virtualisation as we do quite a bit of that stuff at work.

The NAS is still a priority, but at the end of the day it's not production data, it'll just be personal data at home for my media.

I had a look at the ASUS P8B WS motherboard but at almost 300 dollars, it's seems quite steep.

The MSI Z68MA-G45 B3 still looks to be quite the contender at 130 dollars.

Looking at the server build is it just a matter of installing esx and then once i've got esx installed, build VM' for FreeNAS to do the original NAS build?

I'm also guessing that i'd need a hdd for esx to be installed? Or would I be able to put that on a usb stick like freeNAS?

Looks like i've got a fair bit of reading to do.

 

[pitne]

You guys lost me at "run everything on metal"

I think I'm going to go down the vt-d route and learn about esx and virtualisation as we do quite a bit of that stuff at work.

The NAS is still a priority, but at the end of the day it's not production data, it'll just be personal data at home for my media.

I had a look at the ASUS P8B WS motherboard but at almost 300 dollars, it's seems quite steep.

The MSI Z68MA-G45 B3 still looks to be quite the contender at 130 dollars.

Looking at the server build is it just a matter of installing esx and then once i've got esx installed, build VM' for FreeNAS to do the original NAS build?

I have an Asus P8B WS and it works good. Only thing is, it needs unbuffered ecc ram (like all mobo's of this socket) and I want 8GB ECC sticks, and those are expensive right now
I'm also guessing that i'd need a hdd for esx to be installed? Or would I be able to put that on a usb stick like freeNAS?

Looks like i've got a fair bit of reading to do.

You can install ESXi on a USB Stick. However if you do an all-in-one solution (like OpenIndiana + napp-it) you want to install it on hard drive (like a laptop drive or better)

 

[danswartz]

"You guys lost me at "run everything on metal""

"run everything on metal" means on actual HW as opposed to virtualized. i think the vt-d approach is fine.

 

[cantalup]

This is not unique to ZFS. This problem applies to ALL filesystems and storage solutions. The point is that ZFS immediately detects these errors, so it seems that ZFS is fragile when it reports errors. But other storage solutions does not even detect these errors, so you believe everything is ok, which is not. There are numerous stories on this. People change fileystem to ZFS, and suddenly gets error reports. Earlier, these errors where not detected.

as I know in technical terms and my understanding ,
ZFS utilizes a big chunk of memory. ECC is trivial
mdraid linux has a different approach where does not use a big chunk memory. ECC is kind of trivial if the array really big gor RAID5 or 6.

where both are having their advantages .

running 24/7 would bring RAM to drifts, where ECC would help (not much but useful).

I suggest to use ECC for ZFS and mdraid (if really need to make big array 5 or 6).

I had some experience when using on board SATA for mdraid. when one drive is failing, the array randomly frozen until mdraid marked that failed drive and set degraded to the array.
another story was using HBA(using LSI), which gave me smooth transisition to degraded array by really slow access before going to degraded state.

just diving on ZFS for months back(N40L deal), have not seeing a situation when any HD is failing, just wait when that happens in the future. as I understand big/enough memory is the key for ZFS.

 

[cantalup]

You can install ESXi on a USB Stick. However if you do an all-in-one solution (like OpenIndiana + napp-it) you want to install it on hard drive (like a laptop drive or better)

a good point

my Dell T105 running esxi 4.X in USB stick/key , the machine has been running almost 2 years without issue.
I always dumps any OS for development or testing on esxi,
actually, I make many clones for make my life easy when dealing with OS installation in esxi, such as, copy the good backup and transfer to a new image for my development or testing.
those help me saving time and deployments

on my opinion, I always against running NAS (any flavor) in esxi , I prefers to run on a real hardware, where easy to maintain (one of my perspective).

during installing bare metal esxi, I had tried many options for storage media for esxi such as, NFS, and iSCSI. those options are viable but data rate is not good when 2 or more VMs are running.
Just scrapped those Ideas, and slapped internal HD (Hitachi) to T105 where gave me better response(data rate).

all the hardwares are regular desktop mobo, except T105 only
^^^^
those are my opinions that based on my practical life and could be wrong

 

[Balthazar B]

You can install ESXi on a USB Stick. However if you do an all-in-one solution (like OpenIndiana + napp-it) you want to install it on hard drive (like a laptop drive or better)

Hi, pitne. Just to clarify, do you mean you can run ESXi on a USB, but if you run OI by itself on a dedicated NAS (i.e., no virtualization), it should be run from a hard disk? Is it a no-no to run it from an SSD?

 

[danswartz]

Yeah, I'm not sure what he means. There is nothing wrong with having esxi installed on a usb stick, but a local drive (which can be ssd) as the local datastore that OI would need to be on for an all in one.

 

[_Gea]

a good point

on my opinion, I always against running NAS (any flavor) in esxi , I prefers to run on a real hardware, where easy to maintain (one of my perspective).

during installing bare metal esxi, I had tried many options for storage media for esxi such as, NFS, and iSCSI. those options are viable but data rate is not good when 2 or more VMs are running.

It is hard to beat local disks or local Raids in performance.
But it is also true that the "storage features" of ESXi are very basic compared
to a SAN storage server (either comercial or free ZFS variants)

I would never miss options like fast (NFS or SMB) file access to my VM's for backups and cloning,
unlimited and clonable snaps without time delay or initial space consumption, online filechecks against
bitrot, flexible storage pooling and space assignment, storage expandability, accessability
from multiple ESXi machines etc.

For all these, I always need and use a NFS SANServer or a storage server with a SAN-OS like any Solaris flavour,
either as a dedicated machine or virtualized (enough RAM and CPU, vt-d capable, controller and disk pass-through is needed)

If you use a dedicated SAN, you need a fast network with fast switches, best 10 GBe
If you virtualize, you always have fast interconnects between ESXi and storage internally in software.

If you connect ESXi to shared storage via iSCSI or NFS, you must know, that ESXi usually wants to
write all data as sync data in such a case what means that every write must be commited from disk until the next can occur.
(I suppose, this is not the case with ESX and local disks). Sync write can be much slower (up to 100x) than normal writes,
where all writes are going to RAM where they are collected and done optimized after a few seconds.
If you do not need the extra security with this behavour and only need max speed like with local disks,
you can disable this behaviour, in case of ZFS as a property of a NFS shared dataset or you can add a
Log device (RAM or SSD based) to combine security and performance.

 

[mR_CaESaR]

I'm starting to get confused a little more...

Say for example I go ahead with this build right.. (which I know I am, I just have to figure out how I'll be configuring it).

- ESXi installed on a 2.5 laptop hdd (i don't really want to spend $$ on a SSD)
- Once ESX is installed, create a FreeNAS VM
- In the FreeNAS VM, can I then create the 12tb RAIDZ2 array?

Once the array is created in FreeNAS, could I then allocate say 1tb for a "datastore" for all the VMS i then want to create? Leaving the rest for the original plan of having a zfs array for my media?

The more and more I look into this, the more and more it's starting to look overly complicated.

I started with just a windows box with a JBOD setup, I then decided to build another windows JBOD setup as I got pretty much given 15x 1tb wd blacks, but decided to go for FreeNAS, after deciding the h/w for freenas i ended up looking at an esxi build so i can get experience with ESX and so forth...

Like i said, it's all starting to look too complicated for me lol.

 

[cantalup]

It is hard to beat local disks or local Raids in performance.
But it is also true that the "storage features" of ESXi are very basic compared
to a SAN storage server (either comercial or free ZFS variants)

I would never miss options like fast (NFS or SMB) file access to my VM's for backups and cloning,
unlimited and clonable snaps without time delay or initial space consumption, online filechecks against
bitrot, flexible storage pooling and space assignment, storage expandability, accessability
from multiple ESXi machines etc.

For all these, I always need and use a NFS SANServer or a storage server with a SAN-OS like any Solaris flavour,
either as a dedicated machine or virtualized (enough RAM and CPU, vt-d capable, controller and disk pass-through is needed)

If you use a dedicated SAN, you need a fast network with fast switches, best 10 GBe
If you virtualize, you always have fast interconnects between ESXi and storage internally in software.

If you connect ESXi to shared storage via iSCSI or NFS, you must know, that ESXi usually wants to
write all data as sync data in such a case what means that every write must be commited from disk until the next can occur.
(I suppose, this is not the case with ESX and local disks). Sync write can be much slower (up to 100x) than normal writes,
where all writes are going to RAM where they are collected and done optimized after a few seconds.
If you do not need the extra security with this behavour and only need max speed like with local disks,
you can disable this behaviour, in case of ZFS as a property of a NFS shared dataset or you can add a
Log device (RAM or SSD based) to combine security and performance.

thanks for sharing .
I prefers my esxi is stable . I always uses a real hardware for fileserver (ZFS,NFS, and others), since maintenance is localized.
my esxi (4.1 ) is running for testing and development. I prefers not to add extra icing (such as installing fileserver with direct access to local I/O).


I prefer simplicity and planning to deploy openstack ...

Not jumping on SSD till know. still waiting ....

 

[brutalizer]

Regarding running virtualized OS or not. There is a reason in production, you always run on bare metal. Now I am talking about big stock exchange systems, and other critical, large servers. If you dont have such high requirements, virtualized OS is ok to run in production.

mR_CaESaR,
thus for your needs, ESXi will be fine. You are not running a bit stock exchange system at home, with many many users, right? ESXi is very simple too. I suggest you try it. You need to install ESXi on a server, then you use another Windows PC to control the server via a software called vSphere. It is very easy. If you want to install an OS into ESXi, you use that via vSphere. All this software is free. Gea has a ESXi how to, for his nappit. I used that how to, but did not install napp-it. Read it and see how simple ESXi is. Not complicated at all.



For my own needs, I always run Solaris on bare metal. I do agree that it is good practice to install software in virtual machines, which I do. Thus, my Solaris is quite clean and not much software is there. Majority of the software I use, is in different VMs. I use VirtualBox which suffices and is stable for a home user who is not using complicated setups. I have one VM with games. Another VM for work (eclipse, Java, MS Office, Visual Studio, etc) so I have not installed Eclipse into Solaris. Thus, everything is installed into VMs, which I can backup easily.

So I have the same functionality as you ESXi users have, the difference is that I run Solaris bare metal and virtualize everything else via VirtualBox. You virtualize everything - I dont. This means I can also play games and access my zpool at the same time. ESXi might have trouble setting up 3D games and getting raw access to the zpool. Also, I dont need a separate PC to control my Solaris. I have only one PC.

So we reach the same goal in different ways. But I feel safer running Solaris bare metal, than virtualize it. Some people virtualzed Solaris ontop Windows XP via VirtualBox - he lost data because VirtualBox behaved poorly in that early version. But ESXi is tested and well proven to work fine today.

In the end, it is mostly a matter of taste. But, you should use virtualization in some form. Via ESXi or VirtualBox.

 

[mR_CaESaR]

Hey guys, just wondering what your opinion is on a spanner in the works...

What are your thoughts about getting 2x HP Microservers instead of doing a build?

I've just been recommended it and been reading about the microservers and they seem like they are more then capable of doing a few things..

I can set one up purely as a NAS running FreeNAS (5x 2tb in a raidz array).

Another N40L for the VM duties with ESX installed on a USB, 1 drive for the VM datastore, and the other 4 drives in another raidz array with FreeBSD as a VM

You guys think this is a good or bad idea?